grhydro: anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing...

58
GRHydro: A new open source general-relativistic magnetohydrodynamics code for the Einstein Toolkit Philipp Mösta 1 , Bruno C. Mundim 2,3 , Joshua A. Faber 3 , Roland Haas 1,4 , Scott C. Noble 3 , Tanja Bode 8,4 , Frank Löffler 5 , Christian D. Ott 1,5 , Christian Reisswig 1 , Erik Schnetter 6,7,5 1 TAPIR, California Institute of Technology, Pasadena, CA 91125, USA 2 Max-Planck-Institut für Gravitationsphysik, Albert-Einstein-Institut, D-14476 Golm, Germany 3 Center for Computational Relativity and Gravitation and School of Mathematical Sciences, Rochester Institute of Technology, Rochester, NY 14623, USA 4 Center for Relativistic Astrophysics, School of Physics, Georgia Institute of Technology, Atlanta, GA 30332, USA 5 Center for Computation & Technology, Louisiana State University, Baton Rouge, LA 70803, USA 6 Perimeter Institute for Theoretical Physics, Waterloo, ON N2L 2Y5, Canada 7 Department of Physics, University of Guelph, Guelph, ON N1G 2W1, Canada 8 Theoretische Astrophysik, Institut für Astronomie und Astrophysik, Universität Tübingen, 72076 Tübingen, Germany E-mail: [email protected] Abstract. We present the new general-relativistic magnetohydrodynamics (GRMHD) capabilities of the Einstein Toolkit, an open-source community-driven numerical rel- ativity and computational relativistic astrophysics code. The GRMHD extension of the Toolkit builds upon previous releases and implements the evolution of relativis- tic magnetised fluids in the ideal MHD limit in fully dynamical spacetimes using the same shock-capturing techniques previously applied to hydrodynamical evolution. In order to maintain the divergence-free character of the magnetic field, the code imple- ments both hyperbolic divergence cleaning and constrained transport schemes. We present test results for a number of MHD tests in Minkowski and curved spacetimes. Minkowski tests include aligned and oblique planar shocks, cylindrical explosions, mag- netic rotors, Alfvén waves and advected loops, as well as a set of tests designed to study the response of the divergence cleaning scheme to numerically generated monopoles. We study the code’s performance in curved spacetimes with spherical accretion onto a black hole on a fixed background spacetime and in fully dynamical spacetimes by evolutions of a magnetised polytropic neutron star and of the collapse of a magnetised stellar core. Our results agree well with exact solutions where these are available and we demonstrate convergence. All code and input files used to generate the results are available on http://einsteintoolkit.org. This makes our work fully reproducible and provides new users with an introduction to applications of the code. PACS numbers: 04.25.D-, 04.30.-w, 04.70.-s, 07.05.Tp, 95.75.Pq

Upload: others

Post on 27-Sep-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydroA new open source general-relativisticmagnetohydrodynamics code for the Einstein Toolkit

Philipp Moumlsta1 Bruno C Mundim23 Joshua A Faber3Roland Haas14 Scott C Noble3 Tanja Bode84 Frank Loumlffler5Christian D Ott15 Christian Reisswig1 Erik Schnetter675

1 TAPIR California Institute of Technology Pasadena CA 91125 USA2 Max-Planck-Institut fuumlr Gravitationsphysik Albert-Einstein-Institut D-14476Golm Germany3 Center for Computational Relativity and Gravitation and School of MathematicalSciences Rochester Institute of Technology Rochester NY 14623 USA4 Center for Relativistic Astrophysics School of Physics Georgia Institute ofTechnology Atlanta GA 30332 USA5 Center for Computation amp Technology Louisiana State University Baton RougeLA 70803 USA6 Perimeter Institute for Theoretical Physics Waterloo ON N2L 2Y5 Canada7 Department of Physics University of Guelph Guelph ON N1G 2W1 Canada8 Theoretische Astrophysik Institut fuumlr Astronomie und Astrophysik UniversitaumltTuumlbingen 72076 Tuumlbingen Germany

E-mail pmoestatapircaltechedu

Abstract We present the new general-relativistic magnetohydrodynamics (GRMHD)capabilities of the Einstein Toolkit an open-source community-driven numerical rel-ativity and computational relativistic astrophysics code The GRMHD extension ofthe Toolkit builds upon previous releases and implements the evolution of relativis-tic magnetised fluids in the ideal MHD limit in fully dynamical spacetimes using thesame shock-capturing techniques previously applied to hydrodynamical evolution Inorder to maintain the divergence-free character of the magnetic field the code imple-ments both hyperbolic divergence cleaning and constrained transport schemes Wepresent test results for a number of MHD tests in Minkowski and curved spacetimesMinkowski tests include aligned and oblique planar shocks cylindrical explosions mag-netic rotors Alfveacuten waves and advected loops as well as a set of tests designed to studythe response of the divergence cleaning scheme to numerically generated monopolesWe study the codersquos performance in curved spacetimes with spherical accretion ontoa black hole on a fixed background spacetime and in fully dynamical spacetimes byevolutions of a magnetised polytropic neutron star and of the collapse of a magnetisedstellar core Our results agree well with exact solutions where these are available andwe demonstrate convergence All code and input files used to generate the results areavailable on httpeinsteintoolkitorg This makes our work fully reproducibleand provides new users with an introduction to applications of the code

PACS numbers 0425D- 0430-w 0470-s 0705Tp 9575Pq

GRHydro A new open source GRMHD code for the Einstein Toolkit 2

1 Introduction

The recent years have seen rapid developments in the field of numerical relativityBeginning with the first fully general-relativistic (GR) simulations of binary neutronstar (NS) mergers by Shibata and Uryu in 1999 [1] fully dynamical general-relativistichydrodynamics (GRHD) has been explored by a growing number of research groupsA major step forward occurred 2005 when three independent groups developedtwo different techniques the generalised harmonic gauge formalism [2] and the so-called ldquomoving puncturerdquo method [3 4] to evolve vacuum spacetimes containingblack holes (BH) without encountering numerical instabilities that mark the end ofa numerical simulation Since then these evolution schemes have been incorporatedin hydrodynamic simulations and now one or the other is used in essentially every fullGR calculation of merging binaries as well as other astrophysical phenomena involvingdynamical spacetimes such as stellar collapse and BH formation (e [5])

A more recent advance concerns the incorporation of magnetic fields particularlyin the ideal magnetohydrodynamics (MHD) limit into dynamical simulations Buildingupon work that originally focused on astrophysical systems in either Newtonian gravityor fixed GR backgrounds as is appropriate for studies of accretion disks (see [6ndash8] for reviews of the topic) MHD evolution techniques have been incorporated intodynamical GR codes and used to study the evolution of NS-NS (eg [9ndash12]) and BH-NS mergers (eg [13 14]) self-gravitating tori around black holes [15 16] neutron starcollapse (eg [17 18]) stellar core collapse [19] and the evolution of magnetised plasmaaround merging binary BHs (eg [20 21])

Given their rapidly maturing capabilities and widespread use many numericalrelativity codes have been made public either in total or in part The largest community-based effort to do so is the Einstein Toolkit consortium [22 23] which has as one of itsgoals to provide a free publicly available community-driven general-relativistic MHD(GRMHD) code capable of performing simulations that include realistic treatmentsof matter electromagnetic fields and gravity The code is built upon several open-source components that are widely used throughout the numerical relativity communityincluding the Cactus computational infrastructure [24ndash26] the Carpet adaptive meshrefinement (AMR) code [27ndash29] the McLachlan GR evolution code [30 31] and theGRHD code GRHydro [22] which has been developed starting from a public variant of theWhisky code [32ndash35] Many of the details of the Einstein Toolkit may be found in [22]which describes the routines used to provide the supporting computational infrastructurefor grid setup and parallelization constructing initial data evolving dynamical GRHDconfigurations and analysing simulation outputs An extension of the Einstein Toolkitto general multi-patch grids with Cartesian and curvilinear geometry is discussed in [36]

In this paper we describe the newest component of the Einstein Toolkit a MHDevolution scheme incorporated into the GRHydro module Using what is known as theValencia formulation [37ndash39] of the GRMHD equations the new code evolves magneticfields in fully dynamical GR spacetimes under the assumptions of ideal MHD ie the

GRHydro A new open source GRMHD code for the Einstein Toolkit 3

resistivity is taken to be zero and electric fields vanish in the comoving frame of the fluidWhile the evolution equations themselves are easily cast into the same flux-conservativeform as the other hydrodynamics equations the primary challenge for evolving magneticfields is numerically maintaining the divergence-free constraint of the magnetic field Forthis we have implemented a variant of the constrained transport approach [40] and ahyperbolic ldquodivergence cleaningrdquo technique similar to that discussed in [41 42]

The GRHydro module of the Einstein Toolkit is a new independently writtencode andmdashin particularmdashthe development is completely independent of the WhiskyMHDcode [43] since the GRHydro development started from the GRHD (non-MHD) versionof the Whisky code Given its close interaction with the hydrodynamic evolutionthe MHD routines are contained within the GRHydro package (or ldquothornrdquo in thelanguage of Cactus-based routines) It is developed in a public repository alongwith other components of the Einstein Toolkit and it is publicly available under thesame open-source licensing terms as other components in the Toolkit Given thepublic nature of the project the code release includes the subroutines themselvescomplete documentation for their use and parameter files needed to reproduce thetests shown here Code and parameter files are available on the Einstein Toolkit webpage httpeinsteintoolkitorg

In order to validate our GRMHD implementation we include a number of flat-space tests whose solutions are either known exactly or approximately and whichhave frequently been used as a testbed for other GRMHD codes These includeplanar and cylindrical shocktubes a rotating bar threaded by a magnetic field (theldquomagnetic rotorrdquo) propagating Alfveacuten waves and the advection of a flux loop Wetest the codersquos ability to handle static curved spacetime by studying Bondi accretiononto a Schwarzschild BH and perform GRMHD simulations with dynamical spacetimeevolution for a magnetised TOV star and for the collapse of a rotating magnetised stellarcore to a rotating neutron star

This paper is structured as follows in section 2 we describe the Einstein Toolkitand briefly discuss other relativistic MHD codes In section 3 we describe the Valenciaformulation of the GRMHD equations and in section 4 the numerical techniques usedin GRHydro In section 5 we describe the tests we have carried out with the new codeFinally in section 6 we summarise and discuss future directions of the Einstein Toolkit

2 Numerical Relativistic Hydrodynamics and GRMHD codes

The GRHydro code is to the best of our knowledge the first publicly released 3DGRMHD code capable of evolving configurations in fully dynamical spacetimes Stillits development makes use of code and techniques from a number of other public codeswhich we summarise briefly here The most obvious of these is the Einstein Toolkitwithin which its development has taken place As we discuss below the Toolkit itselfis composed of tens of different modules developed by a diverse set of authors whosenumber is approaching 100 The MHD techniques while independently implemented in

GRHydro A new open source GRMHD code for the Einstein Toolkit 4

nearly all cases rely heavily on both the numerical techniques found within the GRHydropackage [22] and thus the original Whisky code [32ndash35] as well as the publicly availableHARM code [44]

21 The Einstein Toolkit

The Einstein Toolkit includes components to aid in performing relativistic astrophysicalsimulations that range from physics modelling (initial conditions evolution analysis)to infrastructure modules (grid setup parallelization IO) and include related tools(workflow management file converters etc) The overall goal is to provide a set ofwell-documented well-tested state-of-the-art components for these tasks while allowingusers to replace these components with their own andor add additional ones

Many components of the Einstein Toolkit use the Cactus ComputationalToolkit [24ndash26] a software framework for high-performance computing Cactussimplifies designing codes in a modular (ldquocomponent-basedrdquo) manner and many existingCactus modules provide infrastructure facilities or basic numerical algorithms such ascoordinates boundary conditions interpolators reduction operators or efficient IO indifferent data formats

The utilities contained in Einstein Toolkit eg help manage components [45 46]build code and submit simulations on supercomputers [47 48] or provide remotedebuggers [49] and post-processing and visualisation interfaces for VisIt [50]

Adaptive mesh refinement (AMR) and multi-block methods are implemented byCarpet [27ndash29] which also provides MPI parallelization Carpet supports Berger-Oligerstyle (ldquoblock-structuredrdquo) adaptive mesh refinement [51] with sub-cycling in time aswell as certain additional features commonly used in numerical relativity (see [27] fordetails) Carpet provides also the respective prolongation (interpolation) and restriction(projection) operators to move data between coarse and fine grids Carpet has beendemonstrated to scale efficiently up to several thousand cores [22]

Carpet supports both vertex-centred and cell-centred AMR In vertex-centredAMR each coarse grid point (cell Center) coincided with a fine grid point (cell Center)which simplifies certain operations (eg restriction and also visualisation) In thispaper we present results obtained with vertex-centred AMR only In cell-centredAMR coarse grid cell faces are aligned with fine grid faces allowing in particular exactconservation across AMR interfaces For a more detailed discussion of cell-centred AMRin hydrodynamics simulations with the Einstein Toolkit we refer the reader to [36]

The evolution of the spacetime metric in the Einstein Toolkit is handled by theMcLachlan package [30 31] This code is autogenerated by Mathematica using theKranc package [52ndash54] implementing the Einstein equations via a 3 + 1-dimensionalsplit using the BSSN formalism [55ndash59] The BSSN equations are finite-differenced at auser-specified order of accuracy and coupling to hydrodynamic variables is included viathe stress-energy tensor The time integration and coupling with curvature are carriedout with the Method of Lines (MoL) [60] implemented in the MoL package

GRHydro A new open source GRMHD code for the Einstein Toolkit 5

Hydrodynamic evolution techniques are provided in the Einstein Toolkit by theGRHydro package a code derived from the public Whisky GRHD code [32ndash35] Thecode is designed to be modular interacting with the vacuum metric evolution onlyby contributions to the stress-energy tensor and by the local values of the metriccomponents and extrinsic curvature It uses a high-resolution shock capturing finite-volume scheme to evolve hydrodynamic quantities with several different methods forreconstruction of states on cell interfaces and Riemann solvers It also assumes anatmosphere to handle low-density and vacuum regions In particular a density floorprevents numerical errors from developing near the edge of matter configurationsBoundaries and symmetries are handled by registering hydrodynamic variables withthe appropriate Cactus routines Passive ldquotracersrdquo or local scalars that are advectedwith the fluid flow may be used though their behaviour is generally unaffected by thepresence or absence of magnetic fields

In adding MHD to the pre-existing hydrodynamics code our design philosophy hasbeen to add the new functionality in such a way that the original code would run withoutalteration should MHD not be required For nearly all of the more complicated routinesin the code this involved creating parallel MHD and non-MHD routines while for someof the more basic routines we simply branch between different code sections dependingon whether MHD is required or not While this requires care when the code is updatedsince some changes may need to be implemented twice it does help to insure that usersperforming non-MHD simulations will be protected against possible errors introducedin the more frequently changing MHD routines

22 Other relativistic MHD codes

In extending the Einstein Toolkit to include MHD functionality we incorporatedtechniques that have previously appeared in the literature Of particular importance isthe HARM code [44 61 62] a free publicly available GRMHD code that can operate onfixed background metrics particularly those describing spherically symmetric or rotatingblack holes Our routines for converting conservative variables back into primitive onesare adapted directly from HARM ([63] see section 44 below) as is the approximatetechnique used to calculate wave speeds for the Riemann solver ([44] see section 43below)

Since many groups have introduced relativistic MHD codes there are a numberof standard tests that may be used to gauge the performance of a particular codeMany of these are compiled in the descriptions of the HARM code mentioned above aswell as papers describing the Athena MHD code [64 65] the Echo GRMHD code [66]the TokyoKyoto grouprsquos GRMHD code [67] the UIUC GRMHD code [68 69] theWhiskyMHD code [43] and the LSU GRMHD code [70] We have implemented several ofthese tests covering all aspects of our code as we describe in section 5 below

GRHydro A new open source GRMHD code for the Einstein Toolkit 6

3 The Valencia formulation of ideal MHD

While GRHydro does not technically assume a particular evolution scheme to evolvethe field equations for the GR metric the most widely used choice for the ET codeis the BSSN formalism [56 57] the particular implementation of which is provided bythe McLachlan code [30 31] We do assume that the metric is known in the ADM(Arnowitt-Deser-Misner) form [71] in which we have

ds2 = gmicroνdxmicrodxν equiv (minusα2 + βiβi)dt2 + 2βidt dxi + γijdxidxj (1)

with gmicroν α βi and γij being the spacetime 4-metric lapse function shift vector andspatial 3-metric respectively Note that we are assuming spacelike signature so theMinkowski metric in flat space reads ηmicroν = diag(minus1 1 1 1) Roman indices are used for3-quantities and 4-quantities are indexed with Greek characters We work in units ofc = G = M = 1 unless explicitly stated otherwise

GRHydro employs the ideal MHD approximation ndash fluids have infinite conductivityand there is no charge separation Thus electric fields Eν = umicroF

muν in the rest frameof the fluid vanishes and ideal MHD corresponds to imposing the following conditions

umicroFmicroν = 0 (2)

Note that throughout this paper we rescale the magnitude of the relativistic Faradaytensor F microν and its dual lowastF microν equiv 1

2εmicroνκλFκλ as well as the magnetic and electric fields by

a factor of 1radic

4π to eliminate the need to include the permittivity and permeability offree space in cgs-Gaussian units

Magnetic fields enter into the equations of hydrodynamics by their contributionsto the stress-energy tensor and through the solutions of Maxwellrsquos equations Thehydrodynamic and electromagnetic contributions to the stress-energy tensor are givenrespectively by

T microνH = ρhumicrouν + Pgmicroν = (ρ+ ρε+ P )umicrouν + Pgmicroν (3)

and

T microνEM = F microλF νλ minus

14g

microνF λκFλκ = b2umicrouν minus bmicrobν + b2

2 gmicroν (4)

where ρ ε P umicro and h equiv 1 + ε + Pρ are the fluid rest mass density specificinternal energy gas pressure 4-velocity and specific enthalpy respectively and bmicro isthe magnetic 4-vector (the projected component of the Maxwell tensor parallel to the4-velocity of the fluid)

bmicro = uνlowastFmicroν

(5)

Note that b2 = bmicrobmicro = 2Pm where Pm is the magnetic pressure Combined the stress-energy tensor takes the form

T microν =(ρ+ ρε+ P + b2

)umicrouν +

(P + b2

2

)gmicroν minus bmicrobν (6)

equiv ρhlowastumicrouν + P lowastgmicroν minus bmicrobν

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 2: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 2

1 Introduction

The recent years have seen rapid developments in the field of numerical relativityBeginning with the first fully general-relativistic (GR) simulations of binary neutronstar (NS) mergers by Shibata and Uryu in 1999 [1] fully dynamical general-relativistichydrodynamics (GRHD) has been explored by a growing number of research groupsA major step forward occurred 2005 when three independent groups developedtwo different techniques the generalised harmonic gauge formalism [2] and the so-called ldquomoving puncturerdquo method [3 4] to evolve vacuum spacetimes containingblack holes (BH) without encountering numerical instabilities that mark the end ofa numerical simulation Since then these evolution schemes have been incorporatedin hydrodynamic simulations and now one or the other is used in essentially every fullGR calculation of merging binaries as well as other astrophysical phenomena involvingdynamical spacetimes such as stellar collapse and BH formation (e [5])

A more recent advance concerns the incorporation of magnetic fields particularlyin the ideal magnetohydrodynamics (MHD) limit into dynamical simulations Buildingupon work that originally focused on astrophysical systems in either Newtonian gravityor fixed GR backgrounds as is appropriate for studies of accretion disks (see [6ndash8] for reviews of the topic) MHD evolution techniques have been incorporated intodynamical GR codes and used to study the evolution of NS-NS (eg [9ndash12]) and BH-NS mergers (eg [13 14]) self-gravitating tori around black holes [15 16] neutron starcollapse (eg [17 18]) stellar core collapse [19] and the evolution of magnetised plasmaaround merging binary BHs (eg [20 21])

Given their rapidly maturing capabilities and widespread use many numericalrelativity codes have been made public either in total or in part The largest community-based effort to do so is the Einstein Toolkit consortium [22 23] which has as one of itsgoals to provide a free publicly available community-driven general-relativistic MHD(GRMHD) code capable of performing simulations that include realistic treatmentsof matter electromagnetic fields and gravity The code is built upon several open-source components that are widely used throughout the numerical relativity communityincluding the Cactus computational infrastructure [24ndash26] the Carpet adaptive meshrefinement (AMR) code [27ndash29] the McLachlan GR evolution code [30 31] and theGRHD code GRHydro [22] which has been developed starting from a public variant of theWhisky code [32ndash35] Many of the details of the Einstein Toolkit may be found in [22]which describes the routines used to provide the supporting computational infrastructurefor grid setup and parallelization constructing initial data evolving dynamical GRHDconfigurations and analysing simulation outputs An extension of the Einstein Toolkitto general multi-patch grids with Cartesian and curvilinear geometry is discussed in [36]

In this paper we describe the newest component of the Einstein Toolkit a MHDevolution scheme incorporated into the GRHydro module Using what is known as theValencia formulation [37ndash39] of the GRMHD equations the new code evolves magneticfields in fully dynamical GR spacetimes under the assumptions of ideal MHD ie the

GRHydro A new open source GRMHD code for the Einstein Toolkit 3

resistivity is taken to be zero and electric fields vanish in the comoving frame of the fluidWhile the evolution equations themselves are easily cast into the same flux-conservativeform as the other hydrodynamics equations the primary challenge for evolving magneticfields is numerically maintaining the divergence-free constraint of the magnetic field Forthis we have implemented a variant of the constrained transport approach [40] and ahyperbolic ldquodivergence cleaningrdquo technique similar to that discussed in [41 42]

The GRHydro module of the Einstein Toolkit is a new independently writtencode andmdashin particularmdashthe development is completely independent of the WhiskyMHDcode [43] since the GRHydro development started from the GRHD (non-MHD) versionof the Whisky code Given its close interaction with the hydrodynamic evolutionthe MHD routines are contained within the GRHydro package (or ldquothornrdquo in thelanguage of Cactus-based routines) It is developed in a public repository alongwith other components of the Einstein Toolkit and it is publicly available under thesame open-source licensing terms as other components in the Toolkit Given thepublic nature of the project the code release includes the subroutines themselvescomplete documentation for their use and parameter files needed to reproduce thetests shown here Code and parameter files are available on the Einstein Toolkit webpage httpeinsteintoolkitorg

In order to validate our GRMHD implementation we include a number of flat-space tests whose solutions are either known exactly or approximately and whichhave frequently been used as a testbed for other GRMHD codes These includeplanar and cylindrical shocktubes a rotating bar threaded by a magnetic field (theldquomagnetic rotorrdquo) propagating Alfveacuten waves and the advection of a flux loop Wetest the codersquos ability to handle static curved spacetime by studying Bondi accretiononto a Schwarzschild BH and perform GRMHD simulations with dynamical spacetimeevolution for a magnetised TOV star and for the collapse of a rotating magnetised stellarcore to a rotating neutron star

This paper is structured as follows in section 2 we describe the Einstein Toolkitand briefly discuss other relativistic MHD codes In section 3 we describe the Valenciaformulation of the GRMHD equations and in section 4 the numerical techniques usedin GRHydro In section 5 we describe the tests we have carried out with the new codeFinally in section 6 we summarise and discuss future directions of the Einstein Toolkit

2 Numerical Relativistic Hydrodynamics and GRMHD codes

The GRHydro code is to the best of our knowledge the first publicly released 3DGRMHD code capable of evolving configurations in fully dynamical spacetimes Stillits development makes use of code and techniques from a number of other public codeswhich we summarise briefly here The most obvious of these is the Einstein Toolkitwithin which its development has taken place As we discuss below the Toolkit itselfis composed of tens of different modules developed by a diverse set of authors whosenumber is approaching 100 The MHD techniques while independently implemented in

GRHydro A new open source GRMHD code for the Einstein Toolkit 4

nearly all cases rely heavily on both the numerical techniques found within the GRHydropackage [22] and thus the original Whisky code [32ndash35] as well as the publicly availableHARM code [44]

21 The Einstein Toolkit

The Einstein Toolkit includes components to aid in performing relativistic astrophysicalsimulations that range from physics modelling (initial conditions evolution analysis)to infrastructure modules (grid setup parallelization IO) and include related tools(workflow management file converters etc) The overall goal is to provide a set ofwell-documented well-tested state-of-the-art components for these tasks while allowingusers to replace these components with their own andor add additional ones

Many components of the Einstein Toolkit use the Cactus ComputationalToolkit [24ndash26] a software framework for high-performance computing Cactussimplifies designing codes in a modular (ldquocomponent-basedrdquo) manner and many existingCactus modules provide infrastructure facilities or basic numerical algorithms such ascoordinates boundary conditions interpolators reduction operators or efficient IO indifferent data formats

The utilities contained in Einstein Toolkit eg help manage components [45 46]build code and submit simulations on supercomputers [47 48] or provide remotedebuggers [49] and post-processing and visualisation interfaces for VisIt [50]

Adaptive mesh refinement (AMR) and multi-block methods are implemented byCarpet [27ndash29] which also provides MPI parallelization Carpet supports Berger-Oligerstyle (ldquoblock-structuredrdquo) adaptive mesh refinement [51] with sub-cycling in time aswell as certain additional features commonly used in numerical relativity (see [27] fordetails) Carpet provides also the respective prolongation (interpolation) and restriction(projection) operators to move data between coarse and fine grids Carpet has beendemonstrated to scale efficiently up to several thousand cores [22]

Carpet supports both vertex-centred and cell-centred AMR In vertex-centredAMR each coarse grid point (cell Center) coincided with a fine grid point (cell Center)which simplifies certain operations (eg restriction and also visualisation) In thispaper we present results obtained with vertex-centred AMR only In cell-centredAMR coarse grid cell faces are aligned with fine grid faces allowing in particular exactconservation across AMR interfaces For a more detailed discussion of cell-centred AMRin hydrodynamics simulations with the Einstein Toolkit we refer the reader to [36]

The evolution of the spacetime metric in the Einstein Toolkit is handled by theMcLachlan package [30 31] This code is autogenerated by Mathematica using theKranc package [52ndash54] implementing the Einstein equations via a 3 + 1-dimensionalsplit using the BSSN formalism [55ndash59] The BSSN equations are finite-differenced at auser-specified order of accuracy and coupling to hydrodynamic variables is included viathe stress-energy tensor The time integration and coupling with curvature are carriedout with the Method of Lines (MoL) [60] implemented in the MoL package

GRHydro A new open source GRMHD code for the Einstein Toolkit 5

Hydrodynamic evolution techniques are provided in the Einstein Toolkit by theGRHydro package a code derived from the public Whisky GRHD code [32ndash35] Thecode is designed to be modular interacting with the vacuum metric evolution onlyby contributions to the stress-energy tensor and by the local values of the metriccomponents and extrinsic curvature It uses a high-resolution shock capturing finite-volume scheme to evolve hydrodynamic quantities with several different methods forreconstruction of states on cell interfaces and Riemann solvers It also assumes anatmosphere to handle low-density and vacuum regions In particular a density floorprevents numerical errors from developing near the edge of matter configurationsBoundaries and symmetries are handled by registering hydrodynamic variables withthe appropriate Cactus routines Passive ldquotracersrdquo or local scalars that are advectedwith the fluid flow may be used though their behaviour is generally unaffected by thepresence or absence of magnetic fields

In adding MHD to the pre-existing hydrodynamics code our design philosophy hasbeen to add the new functionality in such a way that the original code would run withoutalteration should MHD not be required For nearly all of the more complicated routinesin the code this involved creating parallel MHD and non-MHD routines while for someof the more basic routines we simply branch between different code sections dependingon whether MHD is required or not While this requires care when the code is updatedsince some changes may need to be implemented twice it does help to insure that usersperforming non-MHD simulations will be protected against possible errors introducedin the more frequently changing MHD routines

22 Other relativistic MHD codes

In extending the Einstein Toolkit to include MHD functionality we incorporatedtechniques that have previously appeared in the literature Of particular importance isthe HARM code [44 61 62] a free publicly available GRMHD code that can operate onfixed background metrics particularly those describing spherically symmetric or rotatingblack holes Our routines for converting conservative variables back into primitive onesare adapted directly from HARM ([63] see section 44 below) as is the approximatetechnique used to calculate wave speeds for the Riemann solver ([44] see section 43below)

Since many groups have introduced relativistic MHD codes there are a numberof standard tests that may be used to gauge the performance of a particular codeMany of these are compiled in the descriptions of the HARM code mentioned above aswell as papers describing the Athena MHD code [64 65] the Echo GRMHD code [66]the TokyoKyoto grouprsquos GRMHD code [67] the UIUC GRMHD code [68 69] theWhiskyMHD code [43] and the LSU GRMHD code [70] We have implemented several ofthese tests covering all aspects of our code as we describe in section 5 below

GRHydro A new open source GRMHD code for the Einstein Toolkit 6

3 The Valencia formulation of ideal MHD

While GRHydro does not technically assume a particular evolution scheme to evolvethe field equations for the GR metric the most widely used choice for the ET codeis the BSSN formalism [56 57] the particular implementation of which is provided bythe McLachlan code [30 31] We do assume that the metric is known in the ADM(Arnowitt-Deser-Misner) form [71] in which we have

ds2 = gmicroνdxmicrodxν equiv (minusα2 + βiβi)dt2 + 2βidt dxi + γijdxidxj (1)

with gmicroν α βi and γij being the spacetime 4-metric lapse function shift vector andspatial 3-metric respectively Note that we are assuming spacelike signature so theMinkowski metric in flat space reads ηmicroν = diag(minus1 1 1 1) Roman indices are used for3-quantities and 4-quantities are indexed with Greek characters We work in units ofc = G = M = 1 unless explicitly stated otherwise

GRHydro employs the ideal MHD approximation ndash fluids have infinite conductivityand there is no charge separation Thus electric fields Eν = umicroF

muν in the rest frameof the fluid vanishes and ideal MHD corresponds to imposing the following conditions

umicroFmicroν = 0 (2)

Note that throughout this paper we rescale the magnitude of the relativistic Faradaytensor F microν and its dual lowastF microν equiv 1

2εmicroνκλFκλ as well as the magnetic and electric fields by

a factor of 1radic

4π to eliminate the need to include the permittivity and permeability offree space in cgs-Gaussian units

Magnetic fields enter into the equations of hydrodynamics by their contributionsto the stress-energy tensor and through the solutions of Maxwellrsquos equations Thehydrodynamic and electromagnetic contributions to the stress-energy tensor are givenrespectively by

T microνH = ρhumicrouν + Pgmicroν = (ρ+ ρε+ P )umicrouν + Pgmicroν (3)

and

T microνEM = F microλF νλ minus

14g

microνF λκFλκ = b2umicrouν minus bmicrobν + b2

2 gmicroν (4)

where ρ ε P umicro and h equiv 1 + ε + Pρ are the fluid rest mass density specificinternal energy gas pressure 4-velocity and specific enthalpy respectively and bmicro isthe magnetic 4-vector (the projected component of the Maxwell tensor parallel to the4-velocity of the fluid)

bmicro = uνlowastFmicroν

(5)

Note that b2 = bmicrobmicro = 2Pm where Pm is the magnetic pressure Combined the stress-energy tensor takes the form

T microν =(ρ+ ρε+ P + b2

)umicrouν +

(P + b2

2

)gmicroν minus bmicrobν (6)

equiv ρhlowastumicrouν + P lowastgmicroν minus bmicrobν

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 3: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 3

resistivity is taken to be zero and electric fields vanish in the comoving frame of the fluidWhile the evolution equations themselves are easily cast into the same flux-conservativeform as the other hydrodynamics equations the primary challenge for evolving magneticfields is numerically maintaining the divergence-free constraint of the magnetic field Forthis we have implemented a variant of the constrained transport approach [40] and ahyperbolic ldquodivergence cleaningrdquo technique similar to that discussed in [41 42]

The GRHydro module of the Einstein Toolkit is a new independently writtencode andmdashin particularmdashthe development is completely independent of the WhiskyMHDcode [43] since the GRHydro development started from the GRHD (non-MHD) versionof the Whisky code Given its close interaction with the hydrodynamic evolutionthe MHD routines are contained within the GRHydro package (or ldquothornrdquo in thelanguage of Cactus-based routines) It is developed in a public repository alongwith other components of the Einstein Toolkit and it is publicly available under thesame open-source licensing terms as other components in the Toolkit Given thepublic nature of the project the code release includes the subroutines themselvescomplete documentation for their use and parameter files needed to reproduce thetests shown here Code and parameter files are available on the Einstein Toolkit webpage httpeinsteintoolkitorg

In order to validate our GRMHD implementation we include a number of flat-space tests whose solutions are either known exactly or approximately and whichhave frequently been used as a testbed for other GRMHD codes These includeplanar and cylindrical shocktubes a rotating bar threaded by a magnetic field (theldquomagnetic rotorrdquo) propagating Alfveacuten waves and the advection of a flux loop Wetest the codersquos ability to handle static curved spacetime by studying Bondi accretiononto a Schwarzschild BH and perform GRMHD simulations with dynamical spacetimeevolution for a magnetised TOV star and for the collapse of a rotating magnetised stellarcore to a rotating neutron star

This paper is structured as follows in section 2 we describe the Einstein Toolkitand briefly discuss other relativistic MHD codes In section 3 we describe the Valenciaformulation of the GRMHD equations and in section 4 the numerical techniques usedin GRHydro In section 5 we describe the tests we have carried out with the new codeFinally in section 6 we summarise and discuss future directions of the Einstein Toolkit

2 Numerical Relativistic Hydrodynamics and GRMHD codes

The GRHydro code is to the best of our knowledge the first publicly released 3DGRMHD code capable of evolving configurations in fully dynamical spacetimes Stillits development makes use of code and techniques from a number of other public codeswhich we summarise briefly here The most obvious of these is the Einstein Toolkitwithin which its development has taken place As we discuss below the Toolkit itselfis composed of tens of different modules developed by a diverse set of authors whosenumber is approaching 100 The MHD techniques while independently implemented in

GRHydro A new open source GRMHD code for the Einstein Toolkit 4

nearly all cases rely heavily on both the numerical techniques found within the GRHydropackage [22] and thus the original Whisky code [32ndash35] as well as the publicly availableHARM code [44]

21 The Einstein Toolkit

The Einstein Toolkit includes components to aid in performing relativistic astrophysicalsimulations that range from physics modelling (initial conditions evolution analysis)to infrastructure modules (grid setup parallelization IO) and include related tools(workflow management file converters etc) The overall goal is to provide a set ofwell-documented well-tested state-of-the-art components for these tasks while allowingusers to replace these components with their own andor add additional ones

Many components of the Einstein Toolkit use the Cactus ComputationalToolkit [24ndash26] a software framework for high-performance computing Cactussimplifies designing codes in a modular (ldquocomponent-basedrdquo) manner and many existingCactus modules provide infrastructure facilities or basic numerical algorithms such ascoordinates boundary conditions interpolators reduction operators or efficient IO indifferent data formats

The utilities contained in Einstein Toolkit eg help manage components [45 46]build code and submit simulations on supercomputers [47 48] or provide remotedebuggers [49] and post-processing and visualisation interfaces for VisIt [50]

Adaptive mesh refinement (AMR) and multi-block methods are implemented byCarpet [27ndash29] which also provides MPI parallelization Carpet supports Berger-Oligerstyle (ldquoblock-structuredrdquo) adaptive mesh refinement [51] with sub-cycling in time aswell as certain additional features commonly used in numerical relativity (see [27] fordetails) Carpet provides also the respective prolongation (interpolation) and restriction(projection) operators to move data between coarse and fine grids Carpet has beendemonstrated to scale efficiently up to several thousand cores [22]

Carpet supports both vertex-centred and cell-centred AMR In vertex-centredAMR each coarse grid point (cell Center) coincided with a fine grid point (cell Center)which simplifies certain operations (eg restriction and also visualisation) In thispaper we present results obtained with vertex-centred AMR only In cell-centredAMR coarse grid cell faces are aligned with fine grid faces allowing in particular exactconservation across AMR interfaces For a more detailed discussion of cell-centred AMRin hydrodynamics simulations with the Einstein Toolkit we refer the reader to [36]

The evolution of the spacetime metric in the Einstein Toolkit is handled by theMcLachlan package [30 31] This code is autogenerated by Mathematica using theKranc package [52ndash54] implementing the Einstein equations via a 3 + 1-dimensionalsplit using the BSSN formalism [55ndash59] The BSSN equations are finite-differenced at auser-specified order of accuracy and coupling to hydrodynamic variables is included viathe stress-energy tensor The time integration and coupling with curvature are carriedout with the Method of Lines (MoL) [60] implemented in the MoL package

GRHydro A new open source GRMHD code for the Einstein Toolkit 5

Hydrodynamic evolution techniques are provided in the Einstein Toolkit by theGRHydro package a code derived from the public Whisky GRHD code [32ndash35] Thecode is designed to be modular interacting with the vacuum metric evolution onlyby contributions to the stress-energy tensor and by the local values of the metriccomponents and extrinsic curvature It uses a high-resolution shock capturing finite-volume scheme to evolve hydrodynamic quantities with several different methods forreconstruction of states on cell interfaces and Riemann solvers It also assumes anatmosphere to handle low-density and vacuum regions In particular a density floorprevents numerical errors from developing near the edge of matter configurationsBoundaries and symmetries are handled by registering hydrodynamic variables withthe appropriate Cactus routines Passive ldquotracersrdquo or local scalars that are advectedwith the fluid flow may be used though their behaviour is generally unaffected by thepresence or absence of magnetic fields

In adding MHD to the pre-existing hydrodynamics code our design philosophy hasbeen to add the new functionality in such a way that the original code would run withoutalteration should MHD not be required For nearly all of the more complicated routinesin the code this involved creating parallel MHD and non-MHD routines while for someof the more basic routines we simply branch between different code sections dependingon whether MHD is required or not While this requires care when the code is updatedsince some changes may need to be implemented twice it does help to insure that usersperforming non-MHD simulations will be protected against possible errors introducedin the more frequently changing MHD routines

22 Other relativistic MHD codes

In extending the Einstein Toolkit to include MHD functionality we incorporatedtechniques that have previously appeared in the literature Of particular importance isthe HARM code [44 61 62] a free publicly available GRMHD code that can operate onfixed background metrics particularly those describing spherically symmetric or rotatingblack holes Our routines for converting conservative variables back into primitive onesare adapted directly from HARM ([63] see section 44 below) as is the approximatetechnique used to calculate wave speeds for the Riemann solver ([44] see section 43below)

Since many groups have introduced relativistic MHD codes there are a numberof standard tests that may be used to gauge the performance of a particular codeMany of these are compiled in the descriptions of the HARM code mentioned above aswell as papers describing the Athena MHD code [64 65] the Echo GRMHD code [66]the TokyoKyoto grouprsquos GRMHD code [67] the UIUC GRMHD code [68 69] theWhiskyMHD code [43] and the LSU GRMHD code [70] We have implemented several ofthese tests covering all aspects of our code as we describe in section 5 below

GRHydro A new open source GRMHD code for the Einstein Toolkit 6

3 The Valencia formulation of ideal MHD

While GRHydro does not technically assume a particular evolution scheme to evolvethe field equations for the GR metric the most widely used choice for the ET codeis the BSSN formalism [56 57] the particular implementation of which is provided bythe McLachlan code [30 31] We do assume that the metric is known in the ADM(Arnowitt-Deser-Misner) form [71] in which we have

ds2 = gmicroνdxmicrodxν equiv (minusα2 + βiβi)dt2 + 2βidt dxi + γijdxidxj (1)

with gmicroν α βi and γij being the spacetime 4-metric lapse function shift vector andspatial 3-metric respectively Note that we are assuming spacelike signature so theMinkowski metric in flat space reads ηmicroν = diag(minus1 1 1 1) Roman indices are used for3-quantities and 4-quantities are indexed with Greek characters We work in units ofc = G = M = 1 unless explicitly stated otherwise

GRHydro employs the ideal MHD approximation ndash fluids have infinite conductivityand there is no charge separation Thus electric fields Eν = umicroF

muν in the rest frameof the fluid vanishes and ideal MHD corresponds to imposing the following conditions

umicroFmicroν = 0 (2)

Note that throughout this paper we rescale the magnitude of the relativistic Faradaytensor F microν and its dual lowastF microν equiv 1

2εmicroνκλFκλ as well as the magnetic and electric fields by

a factor of 1radic

4π to eliminate the need to include the permittivity and permeability offree space in cgs-Gaussian units

Magnetic fields enter into the equations of hydrodynamics by their contributionsto the stress-energy tensor and through the solutions of Maxwellrsquos equations Thehydrodynamic and electromagnetic contributions to the stress-energy tensor are givenrespectively by

T microνH = ρhumicrouν + Pgmicroν = (ρ+ ρε+ P )umicrouν + Pgmicroν (3)

and

T microνEM = F microλF νλ minus

14g

microνF λκFλκ = b2umicrouν minus bmicrobν + b2

2 gmicroν (4)

where ρ ε P umicro and h equiv 1 + ε + Pρ are the fluid rest mass density specificinternal energy gas pressure 4-velocity and specific enthalpy respectively and bmicro isthe magnetic 4-vector (the projected component of the Maxwell tensor parallel to the4-velocity of the fluid)

bmicro = uνlowastFmicroν

(5)

Note that b2 = bmicrobmicro = 2Pm where Pm is the magnetic pressure Combined the stress-energy tensor takes the form

T microν =(ρ+ ρε+ P + b2

)umicrouν +

(P + b2

2

)gmicroν minus bmicrobν (6)

equiv ρhlowastumicrouν + P lowastgmicroν minus bmicrobν

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 4: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 4

nearly all cases rely heavily on both the numerical techniques found within the GRHydropackage [22] and thus the original Whisky code [32ndash35] as well as the publicly availableHARM code [44]

21 The Einstein Toolkit

The Einstein Toolkit includes components to aid in performing relativistic astrophysicalsimulations that range from physics modelling (initial conditions evolution analysis)to infrastructure modules (grid setup parallelization IO) and include related tools(workflow management file converters etc) The overall goal is to provide a set ofwell-documented well-tested state-of-the-art components for these tasks while allowingusers to replace these components with their own andor add additional ones

Many components of the Einstein Toolkit use the Cactus ComputationalToolkit [24ndash26] a software framework for high-performance computing Cactussimplifies designing codes in a modular (ldquocomponent-basedrdquo) manner and many existingCactus modules provide infrastructure facilities or basic numerical algorithms such ascoordinates boundary conditions interpolators reduction operators or efficient IO indifferent data formats

The utilities contained in Einstein Toolkit eg help manage components [45 46]build code and submit simulations on supercomputers [47 48] or provide remotedebuggers [49] and post-processing and visualisation interfaces for VisIt [50]

Adaptive mesh refinement (AMR) and multi-block methods are implemented byCarpet [27ndash29] which also provides MPI parallelization Carpet supports Berger-Oligerstyle (ldquoblock-structuredrdquo) adaptive mesh refinement [51] with sub-cycling in time aswell as certain additional features commonly used in numerical relativity (see [27] fordetails) Carpet provides also the respective prolongation (interpolation) and restriction(projection) operators to move data between coarse and fine grids Carpet has beendemonstrated to scale efficiently up to several thousand cores [22]

Carpet supports both vertex-centred and cell-centred AMR In vertex-centredAMR each coarse grid point (cell Center) coincided with a fine grid point (cell Center)which simplifies certain operations (eg restriction and also visualisation) In thispaper we present results obtained with vertex-centred AMR only In cell-centredAMR coarse grid cell faces are aligned with fine grid faces allowing in particular exactconservation across AMR interfaces For a more detailed discussion of cell-centred AMRin hydrodynamics simulations with the Einstein Toolkit we refer the reader to [36]

The evolution of the spacetime metric in the Einstein Toolkit is handled by theMcLachlan package [30 31] This code is autogenerated by Mathematica using theKranc package [52ndash54] implementing the Einstein equations via a 3 + 1-dimensionalsplit using the BSSN formalism [55ndash59] The BSSN equations are finite-differenced at auser-specified order of accuracy and coupling to hydrodynamic variables is included viathe stress-energy tensor The time integration and coupling with curvature are carriedout with the Method of Lines (MoL) [60] implemented in the MoL package

GRHydro A new open source GRMHD code for the Einstein Toolkit 5

Hydrodynamic evolution techniques are provided in the Einstein Toolkit by theGRHydro package a code derived from the public Whisky GRHD code [32ndash35] Thecode is designed to be modular interacting with the vacuum metric evolution onlyby contributions to the stress-energy tensor and by the local values of the metriccomponents and extrinsic curvature It uses a high-resolution shock capturing finite-volume scheme to evolve hydrodynamic quantities with several different methods forreconstruction of states on cell interfaces and Riemann solvers It also assumes anatmosphere to handle low-density and vacuum regions In particular a density floorprevents numerical errors from developing near the edge of matter configurationsBoundaries and symmetries are handled by registering hydrodynamic variables withthe appropriate Cactus routines Passive ldquotracersrdquo or local scalars that are advectedwith the fluid flow may be used though their behaviour is generally unaffected by thepresence or absence of magnetic fields

In adding MHD to the pre-existing hydrodynamics code our design philosophy hasbeen to add the new functionality in such a way that the original code would run withoutalteration should MHD not be required For nearly all of the more complicated routinesin the code this involved creating parallel MHD and non-MHD routines while for someof the more basic routines we simply branch between different code sections dependingon whether MHD is required or not While this requires care when the code is updatedsince some changes may need to be implemented twice it does help to insure that usersperforming non-MHD simulations will be protected against possible errors introducedin the more frequently changing MHD routines

22 Other relativistic MHD codes

In extending the Einstein Toolkit to include MHD functionality we incorporatedtechniques that have previously appeared in the literature Of particular importance isthe HARM code [44 61 62] a free publicly available GRMHD code that can operate onfixed background metrics particularly those describing spherically symmetric or rotatingblack holes Our routines for converting conservative variables back into primitive onesare adapted directly from HARM ([63] see section 44 below) as is the approximatetechnique used to calculate wave speeds for the Riemann solver ([44] see section 43below)

Since many groups have introduced relativistic MHD codes there are a numberof standard tests that may be used to gauge the performance of a particular codeMany of these are compiled in the descriptions of the HARM code mentioned above aswell as papers describing the Athena MHD code [64 65] the Echo GRMHD code [66]the TokyoKyoto grouprsquos GRMHD code [67] the UIUC GRMHD code [68 69] theWhiskyMHD code [43] and the LSU GRMHD code [70] We have implemented several ofthese tests covering all aspects of our code as we describe in section 5 below

GRHydro A new open source GRMHD code for the Einstein Toolkit 6

3 The Valencia formulation of ideal MHD

While GRHydro does not technically assume a particular evolution scheme to evolvethe field equations for the GR metric the most widely used choice for the ET codeis the BSSN formalism [56 57] the particular implementation of which is provided bythe McLachlan code [30 31] We do assume that the metric is known in the ADM(Arnowitt-Deser-Misner) form [71] in which we have

ds2 = gmicroνdxmicrodxν equiv (minusα2 + βiβi)dt2 + 2βidt dxi + γijdxidxj (1)

with gmicroν α βi and γij being the spacetime 4-metric lapse function shift vector andspatial 3-metric respectively Note that we are assuming spacelike signature so theMinkowski metric in flat space reads ηmicroν = diag(minus1 1 1 1) Roman indices are used for3-quantities and 4-quantities are indexed with Greek characters We work in units ofc = G = M = 1 unless explicitly stated otherwise

GRHydro employs the ideal MHD approximation ndash fluids have infinite conductivityand there is no charge separation Thus electric fields Eν = umicroF

muν in the rest frameof the fluid vanishes and ideal MHD corresponds to imposing the following conditions

umicroFmicroν = 0 (2)

Note that throughout this paper we rescale the magnitude of the relativistic Faradaytensor F microν and its dual lowastF microν equiv 1

2εmicroνκλFκλ as well as the magnetic and electric fields by

a factor of 1radic

4π to eliminate the need to include the permittivity and permeability offree space in cgs-Gaussian units

Magnetic fields enter into the equations of hydrodynamics by their contributionsto the stress-energy tensor and through the solutions of Maxwellrsquos equations Thehydrodynamic and electromagnetic contributions to the stress-energy tensor are givenrespectively by

T microνH = ρhumicrouν + Pgmicroν = (ρ+ ρε+ P )umicrouν + Pgmicroν (3)

and

T microνEM = F microλF νλ minus

14g

microνF λκFλκ = b2umicrouν minus bmicrobν + b2

2 gmicroν (4)

where ρ ε P umicro and h equiv 1 + ε + Pρ are the fluid rest mass density specificinternal energy gas pressure 4-velocity and specific enthalpy respectively and bmicro isthe magnetic 4-vector (the projected component of the Maxwell tensor parallel to the4-velocity of the fluid)

bmicro = uνlowastFmicroν

(5)

Note that b2 = bmicrobmicro = 2Pm where Pm is the magnetic pressure Combined the stress-energy tensor takes the form

T microν =(ρ+ ρε+ P + b2

)umicrouν +

(P + b2

2

)gmicroν minus bmicrobν (6)

equiv ρhlowastumicrouν + P lowastgmicroν minus bmicrobν

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 5: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 5

Hydrodynamic evolution techniques are provided in the Einstein Toolkit by theGRHydro package a code derived from the public Whisky GRHD code [32ndash35] Thecode is designed to be modular interacting with the vacuum metric evolution onlyby contributions to the stress-energy tensor and by the local values of the metriccomponents and extrinsic curvature It uses a high-resolution shock capturing finite-volume scheme to evolve hydrodynamic quantities with several different methods forreconstruction of states on cell interfaces and Riemann solvers It also assumes anatmosphere to handle low-density and vacuum regions In particular a density floorprevents numerical errors from developing near the edge of matter configurationsBoundaries and symmetries are handled by registering hydrodynamic variables withthe appropriate Cactus routines Passive ldquotracersrdquo or local scalars that are advectedwith the fluid flow may be used though their behaviour is generally unaffected by thepresence or absence of magnetic fields

In adding MHD to the pre-existing hydrodynamics code our design philosophy hasbeen to add the new functionality in such a way that the original code would run withoutalteration should MHD not be required For nearly all of the more complicated routinesin the code this involved creating parallel MHD and non-MHD routines while for someof the more basic routines we simply branch between different code sections dependingon whether MHD is required or not While this requires care when the code is updatedsince some changes may need to be implemented twice it does help to insure that usersperforming non-MHD simulations will be protected against possible errors introducedin the more frequently changing MHD routines

22 Other relativistic MHD codes

In extending the Einstein Toolkit to include MHD functionality we incorporatedtechniques that have previously appeared in the literature Of particular importance isthe HARM code [44 61 62] a free publicly available GRMHD code that can operate onfixed background metrics particularly those describing spherically symmetric or rotatingblack holes Our routines for converting conservative variables back into primitive onesare adapted directly from HARM ([63] see section 44 below) as is the approximatetechnique used to calculate wave speeds for the Riemann solver ([44] see section 43below)

Since many groups have introduced relativistic MHD codes there are a numberof standard tests that may be used to gauge the performance of a particular codeMany of these are compiled in the descriptions of the HARM code mentioned above aswell as papers describing the Athena MHD code [64 65] the Echo GRMHD code [66]the TokyoKyoto grouprsquos GRMHD code [67] the UIUC GRMHD code [68 69] theWhiskyMHD code [43] and the LSU GRMHD code [70] We have implemented several ofthese tests covering all aspects of our code as we describe in section 5 below

GRHydro A new open source GRMHD code for the Einstein Toolkit 6

3 The Valencia formulation of ideal MHD

While GRHydro does not technically assume a particular evolution scheme to evolvethe field equations for the GR metric the most widely used choice for the ET codeis the BSSN formalism [56 57] the particular implementation of which is provided bythe McLachlan code [30 31] We do assume that the metric is known in the ADM(Arnowitt-Deser-Misner) form [71] in which we have

ds2 = gmicroνdxmicrodxν equiv (minusα2 + βiβi)dt2 + 2βidt dxi + γijdxidxj (1)

with gmicroν α βi and γij being the spacetime 4-metric lapse function shift vector andspatial 3-metric respectively Note that we are assuming spacelike signature so theMinkowski metric in flat space reads ηmicroν = diag(minus1 1 1 1) Roman indices are used for3-quantities and 4-quantities are indexed with Greek characters We work in units ofc = G = M = 1 unless explicitly stated otherwise

GRHydro employs the ideal MHD approximation ndash fluids have infinite conductivityand there is no charge separation Thus electric fields Eν = umicroF

muν in the rest frameof the fluid vanishes and ideal MHD corresponds to imposing the following conditions

umicroFmicroν = 0 (2)

Note that throughout this paper we rescale the magnitude of the relativistic Faradaytensor F microν and its dual lowastF microν equiv 1

2εmicroνκλFκλ as well as the magnetic and electric fields by

a factor of 1radic

4π to eliminate the need to include the permittivity and permeability offree space in cgs-Gaussian units

Magnetic fields enter into the equations of hydrodynamics by their contributionsto the stress-energy tensor and through the solutions of Maxwellrsquos equations Thehydrodynamic and electromagnetic contributions to the stress-energy tensor are givenrespectively by

T microνH = ρhumicrouν + Pgmicroν = (ρ+ ρε+ P )umicrouν + Pgmicroν (3)

and

T microνEM = F microλF νλ minus

14g

microνF λκFλκ = b2umicrouν minus bmicrobν + b2

2 gmicroν (4)

where ρ ε P umicro and h equiv 1 + ε + Pρ are the fluid rest mass density specificinternal energy gas pressure 4-velocity and specific enthalpy respectively and bmicro isthe magnetic 4-vector (the projected component of the Maxwell tensor parallel to the4-velocity of the fluid)

bmicro = uνlowastFmicroν

(5)

Note that b2 = bmicrobmicro = 2Pm where Pm is the magnetic pressure Combined the stress-energy tensor takes the form

T microν =(ρ+ ρε+ P + b2

)umicrouν +

(P + b2

2

)gmicroν minus bmicrobν (6)

equiv ρhlowastumicrouν + P lowastgmicroν minus bmicrobν

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 6: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 6

3 The Valencia formulation of ideal MHD

While GRHydro does not technically assume a particular evolution scheme to evolvethe field equations for the GR metric the most widely used choice for the ET codeis the BSSN formalism [56 57] the particular implementation of which is provided bythe McLachlan code [30 31] We do assume that the metric is known in the ADM(Arnowitt-Deser-Misner) form [71] in which we have

ds2 = gmicroνdxmicrodxν equiv (minusα2 + βiβi)dt2 + 2βidt dxi + γijdxidxj (1)

with gmicroν α βi and γij being the spacetime 4-metric lapse function shift vector andspatial 3-metric respectively Note that we are assuming spacelike signature so theMinkowski metric in flat space reads ηmicroν = diag(minus1 1 1 1) Roman indices are used for3-quantities and 4-quantities are indexed with Greek characters We work in units ofc = G = M = 1 unless explicitly stated otherwise

GRHydro employs the ideal MHD approximation ndash fluids have infinite conductivityand there is no charge separation Thus electric fields Eν = umicroF

muν in the rest frameof the fluid vanishes and ideal MHD corresponds to imposing the following conditions

umicroFmicroν = 0 (2)

Note that throughout this paper we rescale the magnitude of the relativistic Faradaytensor F microν and its dual lowastF microν equiv 1

2εmicroνκλFκλ as well as the magnetic and electric fields by

a factor of 1radic

4π to eliminate the need to include the permittivity and permeability offree space in cgs-Gaussian units

Magnetic fields enter into the equations of hydrodynamics by their contributionsto the stress-energy tensor and through the solutions of Maxwellrsquos equations Thehydrodynamic and electromagnetic contributions to the stress-energy tensor are givenrespectively by

T microνH = ρhumicrouν + Pgmicroν = (ρ+ ρε+ P )umicrouν + Pgmicroν (3)

and

T microνEM = F microλF νλ minus

14g

microνF λκFλκ = b2umicrouν minus bmicrobν + b2

2 gmicroν (4)

where ρ ε P umicro and h equiv 1 + ε + Pρ are the fluid rest mass density specificinternal energy gas pressure 4-velocity and specific enthalpy respectively and bmicro isthe magnetic 4-vector (the projected component of the Maxwell tensor parallel to the4-velocity of the fluid)

bmicro = uνlowastFmicroν

(5)

Note that b2 = bmicrobmicro = 2Pm where Pm is the magnetic pressure Combined the stress-energy tensor takes the form

T microν =(ρ+ ρε+ P + b2

)umicrouν +

(P + b2

2

)gmicroν minus bmicrobν (6)

equiv ρhlowastumicrouν + P lowastgmicroν minus bmicrobν

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 7: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 7

where we define the magnetically modified pressure and enthalpy P lowast = P + Pm =P + b22 and hlowast equiv 1 + ε+ (P + b2) ρ respectively

The spatial magnetic field (living on the spacelike 3-hypersurfaces) is defined asthe Eulerian component of the Maxwell tensor

Bi = nmicrolowastFimicro = minusαlowastF i0

(7)

where nmicro = [minusα 0 0 0] is the normal vector to the hypersurfaceThe equations of ideal GRMHD evolved by GRHydro are derived from the local GR

conservation laws of mass and energy-momentum

nablamicroJmicro = 0 nablamicroT microν = 0 (8)

where nablamicro denotes the covariant derivative with respect to the 4-metric and J micro = ρumicro

is the mass current and from Maxwellrsquos equations

nablaνlowastFmicroν = 0 (9)

The GRHydro scheme is written in a first-order hyperbolic flux-conservativeevolution system for the conserved variables D Si τ and Bi defined in terms of theprimitive variables ρ ε vi and Bi such that

D = radicγρW (10)Sj = radicγ

(ρhlowastW 2vj minus αb0bj

) (11)

τ = radicγ(ρhlowastW 2 minus P lowast minus (αb0)2

)minusD (12)

Bk = radicγBk (13)

where γ is the determinant of γij We choose a definition of the 3-velocity vi thatcorresponds to the velocity seen by an Eulerian observer at rest in the current spatial3-hypersurface [72]

vi = ui

W+ βi

α (14)

and W equiv (1minusvivi)minus12 is the Lorentz factor Note that vi Bi Si and βi are 3-vectorsand their indices are raised and lowered with the 3-metric eg vi equiv γijv

jThe evolution scheme used in GRHydro is often referred to as the Valencia

formulation [6 37ndash39] Our notation here most closely follows that found in [73]The evolution system for the conserved variables representing (8) and the spatialcomponents of (9) is

partUpartt

+ partF i

partx i= S (15)

with

U = [DSj τBk]

F i = αtimes

Dv i

Sj vi +radicγP lowastδ ij minus bjBiW

τv i +radicγP lowastvi minus αb0BiWBkvi minus Bivk

(16)

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 8: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 8

S = αradicγ times

0

T microν(partgνjpartxmicrominus Γλ

microνgλj)

α(T micro0 part lnα

partxmicrominus T microνΓ 0

microν

)~0

(17)

Here v i = v i minus βiα and Γλmicroν are the 4-Christoffel symbols

The time component of (9) yields the condition that the magnetic field is divergence-free the ldquono-monopolesrdquo constraint

nabla middotB equiv 1radicγparti(radic

γBi)

= 0 (18)

which also implies

partiBi = 0 (19)

In practice we implement two different methods to actively enforce this constraint Inthe ldquodivergence cleaningrdquo technique we include the ability to modify the magnetic fieldevolution by introducing a new field variable that dissipates away numerical divergenceswhich we discuss in detail in section 451 below An alternative method commonlycalled ldquoconstrained transportrdquo instead carefully constructs a numerical method so thatthe constraint (19) is satisfied to round-off error at the discrete level We discuss thismethod in section 452

4 Numerical Methods

GRHydrorsquos GRMHD code uses the same infrastructure and backend routines as itspure general relativistic hydrodynamics variant [22] In the following we focus on thediscussion of the numerical methods used in the extension to GRMHD

GRHydrorsquos GRMHD code implements reconstruction of fluid and magnetic fieldvariables to cell interfaces for all of the methods present in the original GRHydro codeTVD (total variation diminishing) (eg [74]) PPM (piecewise parabolic method) [75]and ENO (essentially non-oscillatory) [76 77] In addition we added enhanced PPM(ePPM as described in [36 78]) WENO5 (5th order weighted-ENO) [79] and MP5(5th order monotonicity preserving) [80] both in GRMHD and pure GR hydrodynamicssimulations We discuss the different reconstruction methods in section 42 Thecode computes the solution of the local Riemann problems at cell interfaces using theHLLE (Harten-Lax-van Leer-Einfeldt) [81 82] approximate Riemann solver discussedin section 43 We implement the conversion of conserved to primitive variables forarbitrary equations of state (EOS) including polytropic Γ-law hybrid polytropicΓ-law and microphysical finite-temperature EOS We summarise the new methods forGRMHD in section 44 An important aspect of any (GR)MHD scheme is the numericalmethod used to preserve the divergence free constraint GRHydro implements both thehyperbolic divergence cleaning method (eg [41 42]) and a variant of the constrainedtransport method (eg [40 43 83]) which are both discussed in detail in section 45

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 9: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 9

41 Evaluation of magnetic field expressions

The MHD code stores the values of both the primitive B-field vector Bi (Bvec in thecode) and the evolved conservative field Bi (Bcons in the code) in the frame of theEulerian observers For analysis purposes we include options to compute the magneticfield in fluidrsquos rest frame

b0 = WBkvkα

(20)

bi = Bi

W+W (Bkvk)

(vi minus βi

α

) (21)

b2 = BiBi

W 2 + (Bivi)2 (22)

(23)

42 Reconstruction

In a finite-volume scheme one evaluates fluxes at cell faces by solving Riemann problemsinvolving potentially discontinuous hydrodynamic states on either side of the interfaceTo construct these Riemann problems one must first obtain the fluid state on the leftand right sides of the interface The fluid state is known within cells in the form ofcell-averages of the hydrodynamical variables The reconstruction step interpolates thefluid state from cell averaged values to values at cell interfaces without introducingoscillations at shocks and other discontinuities It is possible to reconstruct eitherprimitive or conserved fluid variables however using the former makes it much easierto guarantee physically valid results for which the pressure is positive and the fluidvelocities sub-luminal

By assuming an orthogonal set of coordinates it is possible to reconstruct eachcoordinate direction independently Thus the fluid reconstruction reduces to a one-dimensional problem

We define ULi+12 to be the value of an element of our conservative variable state

vector U on the left side of the face between Ui equiv U(xi y z) and Ui+1 equiv U(xi+1 y z)where xi is the ith point in the x-direction and UR

i+12 the value on the right side of thesame face (coincident in space but with a potentially different value) These quantitiesare computed directly from the primitive values on the face

421 TVD reconstruction For total variation diminishing (TVD) methods we let

ULi+12 = Ui + F (Ui)∆x

2 URi+12 = Ui+1 minus

F (Ui+1)∆x2 (24)

where F (Ui) is a slope-limited gradient function typically determined by the valuesof Ui+1 minus Ui and Ui minus Uiminus1 with a variety of different forms of the slope limiteravailable In practice all try to accomplish the same task of preserving monotonicity andremoving the possibility of spuriously creating local extrema GRHydro includes minmodsuperbee [84] and monotonized central [85] limiters TVD methods are second-order

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 10: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 10

accurate in regions of smooth monotonic flows At extrema and shocks they reduce tofirst order

422 PPM reconstruction The original piecewise parabolic method (oPPM [75]) usesquadratic functions to represent cell-averages from which new states at cell interfaces areconstructed The fluid state is interpolated using a fourth-order polynomial A numberof subsequent limiter steps constrain parabolic profiles and preserve monotonicity so thatno new extrema can form The version implemented in GRHydro includes the steepeningand flattening routines described in the original PPM papers with a simplified flatteningprocedure that allows for fewer ghost points [32 36] The original PPM always reducesto first order near local extrema and shocks The enhanced PPM (ePPM) maintainshigh-order at local extrema that are smooth [36 78]

423 ENO reconstruction Essentially non-oscillatory (ENO) methods use a divideddifferences approach to achieve high-order accuracy via polynomial interpolation [76 77]without reducing the order near extrema or shocks In the third-order case twointerpolation polynomials with different stencil points are used Based on thesmoothness of the field one of the interpolation polynomials is selected

424 WENO reconstruction Weighted essentially non-oscillatory (WENO) recon-struction [79] is an improved algorithm based on the ENO approach The drawbackof ENO methods is that the order of accuracy is not maximal for the available numberof stencil points Furthermore since a single stencil is selected ENO reconstruction isnot continuous In addition the large number of required if statements make ENO meth-ods unnecessary slow WENO reconstruction on the other hand attempts to overcomeall these drawbacks The essential idea is to combine all possible ENO reconstructionstencils by assigning a weight to each stencil The weight is determined by the localsmoothness of the flow When all weights are non-zero the full set of stencil pointsis used and the maximum allowable order of accuracy for a given number of stencilpoints is achieved When the flow becomes less smooth some weights are suppressedand a particular lower order interpolation stencil dominates the reconstruction Usingthe same number of stencil points as third-order ENO WENO is fifth-order accuratewhen the flow is smooth and reduces to third order near shocks and discontinuities

We implement fifth-order WENO reconstruction based on an improved versionpresented in [86] We introduce three interpolation polynomials that approximate UL

i+1from cell-averages of a given quantity Ui

UL1i+12 = 3

8Uiminus2 minus54Uiminus1 + 15

8 Ui (25)

UL2i+12 = minus1

8Uiminus1 + 34Ui minus

38Ui+1 (26)

UL3i+12 = 3

8Ui + 34Ui+1 minus

18Ui+2 (27)

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 11: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 11

Each of the three polynomials yields a third-order accurate approximation of U at thecell-interface By introducing a convex linear combination of the three interpolationpolynomials

ULi+12 = w1UL1

i+12 + w2UL2i+12 + w3UL3

i+12 (28)

where the weights wi satisfy sumiw

i = 1 it is possible to obtain a fifth-orderinterpolation polynomial that spans all five stencil points Uiminus2 Ui+2 The weightswi are computed using so-called smoothness indicators βi In the original WENOalgorithm [79] they are given by

β1 = 13(4u2

iminus2 minus 19uiminus2uiminus1 + 25u2iminus1 + 11uiminus2ui minus 31uiminus1ui + 10u2

i ) (29)

β2 = 13(4u2

iminus1 minus 13uiminus1ui + 13u2i + 5uiminus1ui+1 minus 13uiui+1 + 4u2

i+1) (30)

β3 = 13(10u2

i minus 31uiui+1 + 25u2i+1 + 11uiui+2 minus 19ui+1ui+2 + 4u2

i+2) (31)

Note that in this section only βi refers to the smoothness indicators (29) and not tothe shift vector (1) The weights are then obtained via

wi = wi

w1 + w2 + w3 with wi = γi

(ε+ βi)2 (32)

where γi = 116 58 516 and ε is a small constant to avoid division by zeroUnfortunately the choice of ε is scale dependent and choosing a fixed numberis inappropriate for cases with large variation in scales The improvement madeby [86] overcomes this problem by modifying the smoothness indicators (29)-(31)Following [86] we compute modified smoothness indicators via

βi = βi + ε∣∣∣U2

∣∣∣+ δ (33)

where |U2| is the sum of the U2j in the i-th stencil and δ is the smallest number that the

chosen floating point variable type can hold Now the smoothness indicators dependon the scale of the reconstructed field In our case we set

βi = βi + ε(∣∣∣U2

∣∣∣+ 1) (34)

with ε = 10minus26 and use these values instead of βi in (29)-(31)

425 MP5 reconstruction Monotonicity-preserving fifth-order (MP5) reconstructionis based on a geometric approach to maintain high-order at maxima that are smooth Incontrast to WENO or ENO MP5 makes use of limiters to a high-order reconstructionpolynomial similar to PPM to avoid oscillations at shocks and discontinuities The keyadvantages of MP5 reconstruction are that it preserves monotonicity and accuracy andis fast In practice it compares favourable to enhanced PPM and WENO in terms ofaccuracy (see Appendix B) MP5 reconstruction is carried out in two steps In a firststep a fifth-order polynomial is used to interpolate cell-averages Ui on cell interfacesULi+12

ULi+12 = (2Uiminus2 minus 13Uiminus1 + 47Ui + 27Ui+1 minus 3Ui+2)60 (35)

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 12: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 12

In a second step the interpolated value ULi+12 is limited Whether a limiter is applied

is determined via the following condition First we computeUMP = Ui + minmod(Ui+1 minus Ui α (Ui minus Uiminus1)) (36)

where α is a constant which we set to 40 and where

minmod(x y) = 12 (sign(x) + sign(y)) min(|x| |y|) (37)

A limiter is not applied when(UL

i+12 minus Ui)(ULi+12 minus UMP ) le ε |U | (38)

where ε is a small constant and |U | is the L2 norm of Ui over the stencil pointsUiminus2 Ui+2 Note that the norm factor is not present in the original algorithm Wehave added this to take into account the scale of the reconstructed field Following [80]we set ε = 10minus10 though in some cases it may be necessary to set ε = 0 to avoidoscillations at strong shocks or contact discontinuities such as the surface of a neutronstar

In case (38) does not hold the following limiter algorithm is applied First wecompute the second derivatives

Dminusi = Uiminus2 minus 2Uiminus1 + Ui (39)

D0i = Uiminus1 minus 2Ui + Ui+1 (40)

D+i = Ui minus 2Ui+1 + Ui+2 (41)

Next we computeDM4i+12 = minmod(4D0

i minusD+i 4D+

i minusD0i D

0i D

+i ) (42)

DM4iminus12 = minmod(4D0

i minusDminusi 4Dminus

i minusD0i D

0i D

minusi ) (43)

where the four-argument minmod function is given by

minmod(w x y z) = 18(sign(w) + sign(x))times (44)

|(sign(w) + sign(y))(sign(w) + sign(z))| timesmin(|w| |x| |y| |z|))

We then computeUUL = Ui + α(Ui minus Ui+1) (45)

UAV = 12(Ui + Ui+1) (46)

UMD = UAV minus 12D

M4i+12 (47)

ULC = Ui + 12(Ui minus Uiminus1) + 4

3DM4iminus12 (48)

Using these expressions we computeUmin = max(min(Ui Ui+1 U

MD)min(Ui UUL ULC)) (49)Umax = min(max(Ui Ui+1 U

MD)max(Ui UUL ULC)) (50)(51)

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 13: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 13

Finally a new limited value for the cell interface ULi+12 is obtained via

ULlimitedi+12 = UL

i+12 + minmod(Umin minus ULi+12 Umax minus UL

i+12) (52)

To obtain the reconstructed value at the right interface URiminus12 the values at

Uiminus2 Ui+2 are replaced by the values Ui+2 Uiminus2For a detailed description and derivation of the MP5 reconstruction algorithm we

refer the reader to the original paper [80]

43 Riemann Solver

We implement the Harten-Lax-van Leer-Einfeldt (HLLE) approximate solver [82 87]The more accurate Roe and Marquina solvers require determining the eigenvaluescharacterising the linearizing hydrodynamic evolution scheme which is extremelyresource intensive while not providing a decisive advantage in accuracy In contrastHLLE uses a two-wave approximation to compute the update terms across thediscontinuity at the cell interface With ξminus and ξ+ the most negative and most positivewave speed eigenvalues present on either side of the interface (including magnetic fieldmodes but not those associated with divergence cleaning subsystem as discussed insection 451) the solution state vector U is assumed to take the form

U =

UL if 0 lt ξminus

Ulowast if ξminus lt 0 lt ξ+

UR if 0 gt ξ+

(53)

with the intermediate state Ulowast given by

Ulowast = ξ+UR minus ξminusUL minus F(UR) + F(UL)ξ+ minus ξminus

(54)

The numerical flux along the interface takes the form

F(U) = ξ+F(UL)minus ξminusF(UR) + ξ+ξminus(UR minusUL)ξ+ minus ξminus

(55)

where

ξminus = min(0 ξminus) ξ+ = max(0 ξ+) (56)

We use the flux terms (55) to evolve the hydrodynamic quantities within our Methodof Lines scheme

Our calculations of the wave speeds is approximate following the methods outlinedin [44] to increase the speed of the calculation at the cost of increased diffusivity Themethod overstates the true wavespeeds by no more than a factor of

radic2 and then

only for certain magnetic field and fluid velocity configurations When computing thewavespeeds we replace the full MHD dispersion relation by the approximate quadraticform (see 27 and 28 of [44])

ω2d = k2

d

[v2A + c2

s

(1minus v2

A

c2

)] (57)

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 14: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 14

where we define the wave vector kmicro equiv (minusω ki) the fluid sound speed cs the Alfveacutenvelocity

vA equivradic

b2

ρh+ b2 =radicb2

ρhlowast (58)

the projected wave vector

Kmicro equiv (δνmicro + umicrouν)kν (59)

and the dispersion relation between frequency and (squared) wave number as

ωd = kmicroumicro = minusωu0 + kui (60)

k2d = KmicroK

micro = ω2d + gνσkνkσ (61)

The resulting quadratic may be written

ξ2[W 2(V 2 minus 1)minus V 2

]minus 2ξ

[αW 2vi(V 2 minus 1) + V 2βi

]+ (62)[

(αWvi)2(V 2 minus 1) + V 2(α2γii minus βiβi

)]= 0

where V 2 equiv v2A + c2

s(1minus v2A) and ξ is the resulting wavespeed Note that the indices are

not to be summed over Instead we find different wavespeeds in different directionsWhen divergence cleaning is used to dissipate spurious numerical constraint

violations that appear in the magnetic field the characteristic wavespeeds include twoadditional (luminal) modes of the divergence cleaning subsystem Since the divergencecleaning subsystem decouples from he remainder of the evolution system its wavespeedsmust be handled slightly differently [88 89] which we discuss below in section 451

44 Conservative to Primitive variable transformations

In GRHD converting the conservative variables back to the primitives is a relativelystraightforward task which can be accomplished by inverting (10) ndash (12) with all ofthe magnetic field quantities set to zero GRHydro accomplishes the task througha 1D Newton-Raphson scheme (summarised in section 554 of [22] and describedin detail in the code documentation [90]) The scheme iterates by estimating thefluid pressure determining the density internal energy and the conservative quantitiesgiven this estimate and using those in turn to calculate a new value for the pressureand its residual The method works for any EOS so long as one can calculate thethermodynamic derivatives dPdρ|ε and dPdε|ρ

MHD adds several complications to the inversion although the additional equation(13) is immediately invertible yielding Bi = Biradicγ Instead the primary difficulty isthat bmicro cannot be immediately determined and there is no simple analogue to the GRHDexpressions that allow us to calculate the density easily once the pressure is specified Asa reminder if we consider the (known) values of the undensitised conservative variables

D equiv Dradicγ

= ρW Si equiv Siradicγ

= ρhlowastW 2vi minus αb0bi (63)

τ equiv τradicγ

= ρhlowastW 2 minus P lowast minus (αb0)2 minus D

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 15: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 15

the GRHD system (in which P lowast = P hlowast = h and bmicro = 0) allows us to define

Q equiv τ + D + P = ρhW 2 (64)

and then determine the density as a function of pressure through the relation

ρ =DradicQ2 minus γijSiSj

Q (65)

and thus the Lorentz factor and internal energy as well (see eg [90]) In GRMHDthe most efficient approach for inverting the conservative variable set is often to use amulti-dimensional Newton-Raphson solver with simplifications possible for barotropicEOS for which the internal energy is assumed to be a function of the density onlyeliminating the need to evolve the energy equation Our methods follow very closelythose of [63] particularly the 2D and 1DW solvers they discuss

We define a few auxiliary quantities for use in our numerical calculationsUnfortunately it is impossible to construct a consistent notation that agrees with boththe Einstein Toolkit release paper [22] and the paper that describes the conservative toprimitive variable inversion scheme [63] so we choose to be consistent with the formerhere noting the key differences in Appendix A

From the values of the conservative variables and the metric we construct themomentum density

Smicro equiv minusnνT νmicro = αT 0micro (66)

whose spatial components are given by the relation Si equiv Si and its normal projectionS given by

Smicro = (δνmicro + nmicronν)Sν (67)

Inside the Newton-Raphson scheme we make use of an auxiliary variable Q definedanalogously to (64) by the expression

Q equiv ρhW 2 (68)

Since the inversion methods for general EOS and barotropic ones differ in some of thedetails we discuss each in turn

441 General EOS The 2D Newton-Raphson approach implemented solves thefollowing two equations for the unknown quantities Q and v2 with all other termsknown from the given conserved set

S2 = SiSi = v2(B2 +Q)2 minus (S middotB)2B

2 + 2QQ2 (69)

S middot n = minus(τ + D) = minus B2

2 (1 + v2) + (S middotB)2

2 Qminus2 minusQ+ P (70)

where all dot products are understood as four-dimensional when involving four-vectorsand three-dimensional when not eg S middotB = SmicroBmicro which is equivalent to SiBi sinceBmicro is a three-vector and thus B0 = 0

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 16: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 16

For a polytropicGamma-law EOS where P = (Γminus 1)u and u equiv ρε is the internalenergy density we may calculate P from the conserved variables and the current guessfor Q and v2 by noting that

P = Γminus 1Γ

[(1minus v2)Qminus D

radic1minus v2

] (71)

The iteration updates Q and v2 subject to the consistency conditions that 0 le v2 le 1and Q gt 0 and a post-iteration check is performed to ensure that ε gt 0

For a more general EOS where (71) does not apply and given the structure of theEOS interface in the Einstein Toolkit it is easier to solve for the internal energy densityu and use it as a variable in a three-dimensional Newton scheme along with Q and v2First noting that

P = (1minus v2)Qminus Dradic

1minus v2 minus u (72)

we may rewrite (70) as

S middot n = minusB2

2 (1 + v2) + (S middotB)2

2 Qminus2 minus v2Qminus Dradic

1minus v2 minus u (73)

In the Newton-Raphson steps we first set ρ = Dradic

1minus v2 and solve

u+ P (ρ u) = Q(1minus v2)minus Dradic

1minus v2 (74)

where the left-hand side will in general be a monotonic function of ρ and u alongwith (69) and (73)

Noting that the density depends only on the constant D and the value of v2 withinthe Newton-Raphson scheme the partial derivatives of the pressure with respect to theNewton-Raphson variables are given by

partP

part(v2) =(partP

partρ

)u

dρd(v2) =

(partPpartρ

minus ε

ρ

(partP

partε

dρd(v2) (75)

partP

partu= 1ρ

(partP

partε

(76)

We write these terms in this way since the Einstein Toolkit EOS interface uses thepressure as a function of density and the specific internal energy ε P = P (ρ ε) andcalculates partial derivatives against those variables rather than ρ and u

442 Barotropic EOS For cases where the pressure and internal energy are functionsof the rest mass density only the EOS is barotropic (a polytropic P = KρΓ EOS isa special case of a barotropic EOS) we need a different inversion technique since thequantity S middot n in (70) requires knowledge of τ which is not evolved in these casesInstead the inversion uses (69) only using it to eliminate the variable v2 from theNewton-Raphson scheme by solving for v2(Q)

v2(Q) = Q2S2 + (S middotB)2(B2 + 2Q)Q2(B2 +Q)2 (77)

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 17: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 17

In the special case of a polytropic EOS we proceed by first solving for ρ(Q) throughan independent Newton-Raphson loop over the equation

ρQ = D2(

1 + ΓKρΓminus1

Γminus 1

) (78)

Next we use (77) and the fact that

v2 = ρ2

D2minus 1 (79)

to replace (69) by an expression given only in terms of Q and ρ(Q)

0 = Q2(B2 +Q)2v2 minusQ2(B2 +Q)2v2

= Q2S2 + (S middotB)2(B2 + 2Q)minus(ρ2

D2minus 1

)Q2(B2 +Q)2 (80)

When performing the Newton-Raphson step all quantities in (80) are known a prioriexcept the iteration variable Q and ρ(Q) which depends upon it The loop over Qmakes use of the derivative of (78) given by

dQ= ρ

D2γKρΓminus2 minusQ (81)

45 Divergence-free Constraint Treatment

One of the main difficulties in numerical MHD simulations is the appearance of non-zero divergence of the magnetic field due to numerical errors violations of (18) thatwould be interpreted as magnetic monopoles A number of techniques have beendesigned to combat these including ldquodivergence cleaningrdquo [41 42 91] a method whichintroduces a new field that both damps and advects divergences off the grid ldquoconstrainedtransportrdquo algorithms [40] that balance out fluxes exactly to maintain divergence-freemagnetic fields to round-off accuracy and magnetic vector potential methods that seekto achieve the same result [69 92] Constrained transport methods are often difficultto implement in simulations that employ mesh refinement since maintaining balance atrefinement boundaries is algorithmically complex In the initial GRHydro MHD releasewe implement both divergence cleaning and constrained transport both of which aredetailed below

451 Divergence cleaning Divergence cleaning works by introducing a new fieldvariable that both damps divergences and advects them off the grid through a hyperbolicequation modelled after the telegraph equation driving numerical solutions towardszero divergence Our implementation follows closely that of [41 42] with some minordifferences The new field variable ψ satisfies the evolution equation

nablamicro

(lowastFmicroν + gmicroνψ

)= κnνψ (82)

a modification of Maxwellrsquos equations (9) that reduces to the familiar form as ψ rarr 0Our parameter κ as we note below determines the damping rate of the divergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 18: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 18

cleaning field ψ and incorporates the ratio of the parabolic to the hyperbolic dampingspeed dependence for the scheme often denoted respectively by cp and ch in other works

The evolution equation for ψ is given by the time component of (82) Notingthat [73]

lowastFmicroν = 1

W(umicroBν minus uνBmicro) (83)

the first term on the left-hand-side of (82) yields after some algebra and use of the factthat B0 = minusαF 00 = 0

nablamicrolowastFmicro0 = minus 1

αradicγpartiradicγBi (84)

and the second termnablamicrog

micro0ψ = gmicro0partmicroψ = 1α2

[minusparttψ + βipartiψ

] (85)

Combining the two we find

parttψ + parti(αBi minus ψβi

)= ψ

(minusκαminus partiβi

)+radicγBiparti

(αradicγ

) (86)

where we have grouped the derivative terms to respect the flux-conservative formThe spatial part of (82) is the modified evolution equation for the magnetic field

The two terms on the left hand side yield respectively

nablamicrolowastFmicroj = 1

αradicγ

parttradicγBj + parti

radicγ[(αvi minus βi

)Bj minus

(αvj minus βj

)Bi]

(87)

nablamicrogmicrojψ = gmicrojpartmicroψ = 1

α2

[βjparttψ +

(α2γij minus βiβj

)partiψ

] (88)

Using (86) to eliminate the parttψ term switching over to Bi as the primary magnetic fieldvariable and combining derivatives to produce a flux-conservative form we find aftersome algebra

parttBj + parti[(αvi minus βi)Bj minus αvjBi + α

radicγγijψ

]= (89)

minus Bipartiβj + ψparti(αradicγγij

)

which reduces to the standard evolution equation for ψ rarr 0 and partiBi rarr 0 To evaluatethe right-hand-side we make use of the identity

parti(radicγγij) = minusradicγγklΓjkl = radicγ

[12γ

ijγklpartiγkl minus γjkγilpartiγkl] (90)

The characteristic velocities of the evolution system for MHD without and withdivergence cleaning differ In the former the largest-magnitude wave speed is thefast magnetosonic wave speed The inclusion of divergence cleaning introduces twoadditional modes corresponding to eigenvalues for the evolution of the divergencecleaning field ψ and the longitudinal component of the magnetic field (ie the casewhere i = j inside the term in brackets in (90)) that decouple from the remaining seveneigenvalues of the system [88 89] In our scheme both modes have characteristic speedequal to the speed of light

vdc = plusmnradicgii + βi

α (91)

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 19: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 19

z

Ez

Ey

x

y

(ijk)

(i+ 12 j- 12 k)

Ey

Ez

BxBx

Figure 1 Sketch of a simulation cell showing the location of face centred magneticfield and edge centred electric fields

Our system which can be written in relativistically covariant form corresponds tosetting the hyperbolic damping speed ch of the scheme to be the speed of light inthe notation of Newtonian divergence cleaning methods [91] This is in accordancewith standard practice in both Newtonian and relativistic calculations to choose acharacteristic speed for divergence cleaning that is as fast as allowed without violatingcausality or in the case of Newtonian calculations the Courant condition We notethat for the luminal-speed modes the HLLE flux formula (55) reduces to the localLax-Friedrichs form when evaluated on a Minkowski background

F (U) = 12[F (UL) + F (UR)minus ξ(UR minus UL)

] (92)

where ξ = max(ξ+ |ξminus|) Test runs performed with the HLLE Riemann solverbut without the luminal-speed velocities often develop large spurious oscillationsparticularly in cases where strong rarefactions are present eg the cylindrical blastwave evolution described in section 53 and the rotor test described in section 54

452 Constrained transport As an alternative to divergence cleaning and to simplifycomparison of the performance of the new code with existing GRMHD codes we alsoimplemented a variant of a constrained transport scheme [93] In constrained transportschemes one carefully constructs a numerical update scheme such that the divergence freeconstraint (19) is conserved to numerical round-off accuracy Rather than implementingthe original scheme proposed in [93ndash95] which relies on a complicated staggering of themagnetic field components we employ the simplified scheme called ldquoflux-CTrdquo describedin [40 43 83] a two-dimensional version of which is used in the HARM code [44] Thescheme uses cell-centred values of the magnetic field For completeness of presentationwe reproduce the basic description of the scheme found in [43] but refer the reader tothe original literature for more details

In constrained transport schemes the induction equation (90) is written in terms ofthe electric field ~E at the edges of each face of a simulation cell (see figure 1)

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 20: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 20

Employing Ampegraverersquos law the time derivative of the face-averaged magnetic fieldcomponent Bx is given by

partBxi+ 1

2 jk

partt∆y∆z = minus Ey

i+ 12 jkminus 1

2∆y minus Ezi+ 1

2 j+12 k

∆z + Eyi+ 1

2 jkminus 12

∆y (93)+ Ezi+ 1

2 jminus12 k

∆z

with analogous equations for the other magnetic field components In the ideal MHDapproximation the electric field components in (94) can be expressed in terms of thefluxes Bkvi minus Bivk for the magnetic field ~B given in (16) Specifically we use thenumerical fluxes of the induction equation to calculate these electric field components

Since the scheme of [43] evolves the cell-averaged magnetic field values it computesthe change in the cell-centred fields as the average change in the face centred fields

partBxijkpartt

= 12

partBximinus 12 jk

partt+partBx

i+ 12 jk

partt

(94)

which is the actual evolution equation implemented in the codeIn this formalism the conserved divergence operator is given by

(nabla middot B)i+ 12 j+

12 k+ 1

2= 1

4∆x

j+1sumjprime=j

k+1sumkprime=k

(Bxi+1jprimekprime minus Bxijprimekprime

)

+ 14∆y

i+1sumiprime=i

k+1sumkprime=k

(Byiprimej+1kprime minus B

yiprimejkprime

)

+ 14∆z

i+1sumiprime=i

j+1sumjprime=j

(Bziprimejprimek+1 minus Bziprimejprimek

) (95)

which can be interpreted as a cell corner-centred definition of a divergence operator

5 Tests

51 Monopole tests

In order to test the divergence cleaning formalism and to explore its properties weimplement a number of tests that initialise the MHD configuration with numericalmonopoles Each case uses a uniform-density uniform-pressure fluid which is initiallyat rest and we assume Minkowski spacetime The ambient magnetic field is assumed tobe zero The initial magnetic configurations include ldquopointrdquo monopoles where Bx 6= 0is set at a single point in the Center of the domain and Gaussian monopoles for which

Bx =eminusr2R2

G minus eminus1 r lt RG

0 r ge RG (96)

where RG is the radius of the compactly supported monopole In the supplementarymaterial we present additional results on the performance of the algorithm when dealingwith high frequency constraint violations

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 21: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 21

In Figs 2 we show the evolution of Gaussian and 3-dimensional alternatingGaussian monopole initial data In each case the grid is chosen to be 2003 spanninga coordinate range from minus2 to 2 in each dimension and we set RG = 02 Thesimulations are performed using second-order Runge-Kutta (RK2) time integrationand TVD-based reconstruction with an monotonized central limiter and a Courant-Friedrichs-Lewy (CFL) factor of 025 (ie ∆t∆x = 025 here) The fluid assumedto follow a Γ = 53 ideal-gas law ie to follow the condition P = (Γ minus 1)ρε was setto an initially stationary state with density and internal energy given by ρ = 10 andε = 01 respectively We vary the divergence cleaning parameter κ that appears in thedriving term in (82) choosing values κ = 1 10 and 100 There is a markedly differentevolution of the divergence of the magnetic field in time following a predictable pattern

The choice κ = 1 yields the slowest damping rate which allows the wave-likebehaviour of the divergence cleaning field to radiate away the divergence of the B-fieldaway from its original location as it damps downward By contrast κ = 100 yields amuch stiffer system of equations resulting in a relatively slow damping rate particularlyfor the lower-frequency terms in the initial data with very low divergence in the magneticfield spreading through the numerical grid The intermediate case κ = 10 yields theclosest analogue to a critically damped system in that the amplitude of the divergencedecreases most rapidly while errors are efficiently radiated away across the grid Ingeneral we would recommend values of κ sim 10 to be used as a default

52 Planar MHD Shocktubes

Historically the simplest tests for an MHD scheme are shocktubes on a static flat-spaceMinkowski metric They are important to demonstrate the ability of the new code tocapture a variety of MHD wave structures Despite being simple setups they do providea stringent tests for the algorithm For our one-dimensional shocktube tests we set upldquoLeftrdquo and ldquoRightrdquo MHD states on either side of a planar interface with parametersdrawn from the five cases considered in [96] and summarised in Table 1 These havebeen widely used by several groups to establish the validity of numerical MHD codesThese parameter choices include generalisations of familiar cases long used to test non-relativistic MHD codes eg case ldquoBalsara1rdquo which is a generalisation of the Brio-Wushock tube problem [97]

To validate the new code we evolve each of these shocks in each of the coordinatedirections (ie ldquoLeftrdquo states for domains of the grid satisfying x lt 0 y lt 0 andz lt 0 and ldquoRightrdquo states for x gt 0 y gt 0 and z gt 0 respectively) finding excellentagreement (limited only by numerical precision) in each direction as expected Wealso evolve shocks with mid-planes oriented obliquely to the coordinate planes choosingx + y = 0 (ldquo2D diagonalrdquo) and x + y + z = 0 (ldquo3-d diagonalrdquo) as the mid-planesWhereas shocks in any of the coordinate planes initially satisfy the divergence freeconstraint to roundoff error and maintain this state indefinitely given the symmetryin the setup diagonal shocks yield non-zero divergence of the magnetic field across the

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 22: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 22

minus2 minus1 0 1 2minus2

minus1

0

1

2

nablamiddotB

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure 2 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a magnetic field with an initially Gaussian Bx profile ofradius RG = 02 We show results for different values of the divergence cleaningdamping parameter κ For κ = 100 (dot-dashed blue curve) the system is quite stiffand damping is substantially slower than in other cases and more of the divergenceremains localised near the initial position while for κ = 1 (dashed red curve) thedamping is only slightly faster but spreads more rapidly away across the grid Forκ = 10 (solid green curve) we find a nearly ideal choice of rapid damping and advectionof the error away from the source and recommend this value as a default for generalcalculations

shock front due to finite resolution effects The simplest way to understand this is torealise that for a 1-dimensional domain (say along the x direction) the x componentof the magnetic field Bx is necessarily constant in space and time as a consequence ofthe induction equation (90) The slab symmetry of the test on the other hand impliesthat all fields in particular the By and Bz field are independent of y and z Thesetwo effects imply that partiBi = 0 to roundoff precision initially and for all times sincethe finite volume scheme preserves this property Diagonal shocks on the other hand

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 23: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 23

break this symmetry by having field components depend on two coordinates introducingdivergence constraint violations of order of the truncation error For non-flat geometriesthe non-linear coupling of GRMHD equations would generate violations on the orderof the truncation error in all directions This multidimensional test allows us to verifythat these constraint violations are controlled by employing divergence cleaning

In all cases the coordinate origin was chosen so that no grid cell was centred directlyon the shock front so that each cell was unambiguously located in the left or right statesIn each run we used RK2 time integration with a CFL factor of 08 for 1-d shocks and04 for 2D diagonal cases with ∆x = 11600 and ∆x = ∆y = 1(800

radic2) for the 1-d

and 2D diagonal cases to ensure equal resolution in the shock direction For all resultsshown below we use the HLLE Riemann solver and TVD reconstruction of the primitivevariables with a monotonized central limiter Boundary conditions are implemented byldquocopyingrdquo all data to grid edges from the nearest points in the interior located on planesparallel to the shock front Divergence cleaning is turned on for the 2D case with κ = 10Table 1 lists the parameters describing each shock test that is performed and we notethat all of them assume an ideal-gas law EOS with the given value of Γ here Γ = 2for the ldquoBalsara1rdquo shock test and Γ = 53 for the others These tests are also includedas options in the code release within the GRHydro_InitData thorn Exact solutions foreach case are computed using the open-source available code of [98 99] In all caseswe see very good agreement with the expected results that are comparable to resultsof similar codes in the literature [43 66 70 73 96 98 100 101] We conclude thatdivergence cleaning does not interfere with generating physically meaningful GRMHDevolution results in the presence of shocks and that the new code performs as well asother codes based on the same numerical techniques [89]

In figure 3 we show results for the relativistic generalisation of the Brio amp Wushock test develop in [96] The initial shock develops into a left-going fast rarefactiona left-going compound wave a contact discontinuity a right-going slow shock and aright-going fast rarefaction We find that the code captures all elementary waves andis in good agreement with the exact solution of [43] In panel (g) of 3 we show theconstraint violation as measure by a 2nd order centred finite difference stencil for partiBiFigure 4 displays the results of the relativistic MHD collision problem (problem 4 of [96]in which two very relativistic (Lorentz factor 2237) streams collide with each other Wefind deviations from the exact solution that are on the same level as in [43 96]

53 Cylindrical Shocks

A more stringent multidimensional code test is provided by a cylindrical blast waveexpanding outward in two dimensions We take the parameters for this test problemfrom [102] The density profile is determined by two radial parameters rin and rout and

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 24: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 24

0204060810

(a)

ρ

Exact solution1D numerical2D numerical

0204060810

(b)

P

0

01

02

03

04

(c)

vx

minus06

minus04

minus02

0(d)

vy

101112131415

(e)

W

minus04 minus02 00 02 04minus10minus05

000510

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 1 Test

Figure 3 Evolution of the Balsara1 shocktube performed with divergence cleaningshowing 1D and 2D diagonal cases as solid (blue) and dashed (red) curves respectivelyFrom top to bottom we show the density ρ gas pressure P normal and tangentialvelocities vx and vy the Lorentz factor W the tangential magnetic field componentBy and the numerical divergence of the magnetic field for the 2D case (for 1D shocksthe divergence is uniformly zero by construction) Parameters for the initial shockconfiguration are given in Table 1 All results are presented at t = Tref given by thefinal column of Table 1 The results agree well with those reported in [43 96] withmost plots being indistinguishable and only the Lorentz factor plot showing a slightovershoot on the left-hand side of the shock A more detailed description of the testsetup and parameters can be found in the main text in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 25: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 25

0

20

40

60(a)

ρ

0

500

1000 (b)

P Exact solution1D numerical2D numerical

minus100minus050

005

1(c)

vx

minus03

minus02

minus01

0(d)

vy

05

101520

(e)

W

minus04 minus02 00 02 04

minus10

0

10

(f)By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 4 Test

Figure 4 Evolution of the Balsara4 shocktube case performed with divergencecleaning with all conventions as in figure 3 The results code reproduces the exactresult very well with deviations from the exact solution similar to those found inn [96]A more detailed description of the test setup and parameters can be found in the maintext in section 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 26: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 26

Table 1 Shock tube parameters For each of these shock tube tests originallycompiled in [96] we list for both the left ldquoLrdquo and right ldquoRrdquo states the fluid density ρspecific internal energy ε fluid 3-velocity ~v equiv vi and tangential magnetic field Bt aswell as the (uniform) normal magnetic field magnitude Bn the adiabatic index Γ ofthe ideal-gas law equation of state and the time Tref at which results are plotted Ourvalues are quoted for shocks moving in the x-direction ie the shock front is orientedin the yminus z plane so Bt is equivalent to (By Bz) and Bn = Bx For diagonal shocksall quantities are rotated appropriately along the diagonal of xminus y plane Results forcases marked by an asterisklowast are shown in the supplementary material only

Name ρL εL ~vL BtL ρR εR ~vR

Balsara1 10 10 ~0 (100) 0125 08 ~0Balsara2lowast 10 450 ~0 (6060) 10 15 ~0Balsara3lowast 10 15000 ~0 (7070) 10 015 ~0Balsara4 10 015 (0999 0 0) (7070) 10 015 (-099900)Balsara5lowast 108 1425 (04 03 02) (0303) 10 15 (-045-0202)

Name BtR Bn Γ Tref

Balsara1 (continued) (-100) 05 2 04Balsara2lowast (continued) (0707) 50 53 04Balsara3lowast (continued) (0707) 100 53 04Balsara4 (continued) (-70-70) 100 53 04Balsara5lowast (continued) (-0705) 20 53 055

two reference states such that

ρ(r) =

ρin r le rin

exp[

(routminusR) ln ρout+(rminusrin) ln ρinroutminusrin

] rin lt r lt rout

ρout r ge rout

(97)

with an equivalent form for the pressure gradient The initial fluid velocity is set tozero and the initial magnetic is uniform in the domain We use the following shockparameters

rin = 08 rout = 10 ρin = 10minus2 ρout = 10minus4 Pin = 10 (98)Pout = 3times 10minus5 Bi = (01 0 0)

All tests use Γ-law equation of state with adiabatic index Γ = 43 We use a 200times200times8grid spanning the coordinate range [minus6 6] in the x and y-directions which is the setupused in [102] To verify that our results are indeed convergent with resolution andto compare to [69] we run two more simulations using 400 and 500 points as well Alltests shown use the divergence cleaning scheme with κ = 5 HLLE Riemann solver TVDreconstruction using a monotonized central limiter No explicit dissipation is added tothe system of equations

This test is known to push the limits of many combinations of MHD evolutiontechniques and often fails for specific combinations of reconstruction methods andRiemann solvers [67 69] In our tests we only explore a small range of settings we

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 27: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 27

verify that the test succeeds when we swap TVD reconstruction with a 2nd order ENOscheme or use constraint transport instead of divergence cleaning In the later case weadd Kreiss-Oliger dissipation [103] of order 3 and strength parameter ε = 3 [103] to themagnetic field variables to stabilise the system

We find this test to be among the most sensitive one of our tests Including thetwo light-like modes of the divergence cleaning field (see section 451) is crucial toobtain correct results When these are not included the code crashes quickly dueto the growth of spurious oscillations in the magnetic field in the rarefaction regionthat develops behind the shock particularly along the diagonals where the shock frontexpands obliquely to the grid (as was seen in tests of a several other codes [66 67 69])We note that implementing the local Lax-Friedrichs flux (see (92)) instead of HLLE alsostabilises the evolution

Two-dimensional profiles are shown in figure 5 for P WLorentz Bx and By along

with numerically determined magnetic field lines at t = 4 These results agree wellwith those shown in figure 10 of [102] One-dimensional slices along the xminus and yminusaxesfor the rest mass density gas pressure magnetic pressure and Lorentz factor at t = 4are shown in figure 6 for three different numerical resolutions Our results show goodqualitative agreement with those presented previously by [67] and particularly by [69]once one accounts for rescalings associated with different choices for the initial shockparameters We find that the benefit of increasing the resolution is largest in the regionwithin the shock front which is consistent with results obtained by other groups [67 69]

54 Magnetic rotor

As a second two-dimensional test problem we simulate the magnetic rotor test firstdescribed in [95] for classical MHD and later generalised to relativistic MHD in [101]The setup consists of a cylindrical column (the rotor) of radius rin = 01 and densityρin = 10 embedded in a medium of lower density ρout = 10 Initially the pressure insidethe column and in the medium are equal Pin = Pout = 1 and the cylinder rotates withuniform angular velocity of Ω = 995 along the cylinder axis so that the fluid 3-velocityreaches a maximum value of vmax = 0995 at the outer edge of the cylinder The fluidoutside the cylinder is initially at rest The equation of state used is a Γ-law withΓ = 53 The cylinder is threaded by an initially uniform magnetic field of magnitudeBx = 10 along the x direction covering the entire space cylinder and exterior Weuse TVD reconstruction with the minmod limiter the HLLE Riemann solver RK2time stepping with CFL factor 025 The magnetic field is evolved using the divergencecleaning technique using a damping factor κ = 50

At the beginning of the simulation a strong discontinuity is present at the edge ofthe cylinder since we do not apply any smoothing there During the simulation magneticbraking slows down the rotor while the magnetic field lines themselves are dragged fromtheir initial horizontal orientation At the end of the simulation at t = 04 the fieldlines in the central region have rotated by nearly 90 degrees while at large radii the

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 28: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 28

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Pressure

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Lorentz factorMagnetic fieldlines

-60 -36 -12 12 36 60-60

-36

-12

12

36

60Bx

-60 -36 -12 12 36 60-60

-36

-12

12

36

60By

Ye

10minus5 10minus3 10minus1 100 219 338 457

000 012 024 036 minus018 minus006 006 018

x

y

x

y

x

y

x

y

Figure 5 Evolved state of the cylindrical explosion test problem with parametersbased on the test presented in [102] At t = 4 we show the following quantitiestop panels Gas pressure P and Lorentz factor W along with numerically determinedmagnetic field lines bottom panels Bx and By The numerical resolution is ∆x = 006Profiles are almost indistinguishable from the solutions found in figure 10 of of [102]The very low pressure region outside of the shock from is slightly more extendedin [102] It is worth noting that [102] apply some numerical resistivity to controlartifacts while our divergence cleaning scheme has dissipative terms present in thedivergence cleaning equations but do not add explicit resistivity hence our results inlow pressure regions might well differ slightly

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 29: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 29

3

6

9

(a)

Cylindrical explosion

ρtimes

104

(b)

10minus5

10minus4

10minus3

10minus2 (c)

P

(d)

10minus5

10minus4

10minus3

10minus2

(e)

b2 2

(f)

minus6 minus3 0 31

2

3

4

5(g)

W

xminus6 minus3 0 3 6

(h)

y

N = 200N = 400N = 500

Figure 6 One-dimensional slices along the xminus and yminusaxes for the evolved state of thecylindrical explosion test Displayed are the rest mass density gas pressure magneticpressure and Lorentz factor at t = 4 The red solid lines corresponds to a resolution of∆x = 006 the black dashed one corresponds to a resolution of ∆x = 003 and the bluedotted on corresponds to a resolution of ∆x = 0024 These results may be comparedto figure 7 of [67] and particularly figure 7 of [69] noting that their parameters differfrom ours which were chosen to match the test as presented by [102] particularly bythe presence of a narrow transition region whose effects are visible as a secondary shockin the outer edges of the shock front Comparing the plots both codes agree on theoverall structure of the result with the two-dimensional pseudocolor plots in 5 beingindistinguishable and the one-dimensional slices clearly showing the same structure ofthe shocks A more detailed discussion of the test setup and results can be found inthe text

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 30: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 30

orientation of the field lines remains unchangedOur results shown in figure 7 for a grid spacing ∆x = 1400 compare well with

those presented in figure 5 of [101] and figure 8 of [69] The density profile at the endof simulation is reproduced showing slightly less noisy behaviour at the location ofthe expanding high density shell than the results reported in [101] In addition themagnetic field lines displayed in figure 5 of [101] are also reproduced very well A slightover-density near the left and right corners of the (now low density) rotor is present inour simulations that is absent in [101]

Figure 8 depicts one-dimensional slices through the simulation domain at the endof the run We show results for three different resolution using 250 400 and 500points to cover the domain We find excellent agreement with previous work at similarresolutions [69]

55 Alfveacuten Wave

The propagation of low-amplitude circularly-polarised Alfveacuten waves has an exactsolution that is useful for testing the performance of the MHD sector of the code [40]and provides a scenario for clean convergence tests since the solution is smooth Theinitial configuration is a uniform-density fluid with a velocity profile given by

vx = 0 vy = minusvAA0 cos(kx) vz = minusvAA0 sin(kx) (99)

and corresponding magnetic field configuration

Bx = const By = A0Bx cos(kx) Bz = A0B

x sin(kx) (100)

where we choose Bx = 10 an amplitude parameter A0 = 10 ρ = 10 A Γ-law EOSwith Γ = 53 was used as well as TVD as reconstruction method a 2nd order Runge-Kutta (RK) [104 105] time integration with the Courant factor set to 02 and the HLLERiemann solver as well as divergence cleaning

The Alfveacuten speed is given by the expression

v2A = 2(Bx)2

ρh+ (Bx)2(1 + A20)

1 +

radicradicradicradic1minus(

2A0(Bx)2

ρh+ (Bx)2(1 + A20)

)2

minus1

(101)

and the wavevector k = 2πLx for one-dimensional cases For two-dimensional caseswe rotate the coordinates so that wave fronts lie along diagonals of the grid In allcases periodic boundary conditions are assumed Several values for the pressure havebeen used for testing purposes by other groups spanning the range P = 01 [40] toP = 10 [106] but here we choose P = 05 which yields the convenient result vA = 05In time we expect the wave to propagate across the grid such that if it travels aninteger number of wavelengths we should in the ideal case reproduce our initial dataexactly In figure 9 we show the convergence results for our code for both 1-dimensionaland 2-dimensional cases finding the expected second-order convergence with numericalresolution when plotting the L2 norm of the difference between the evolved solution andthe initial data evaluated along the xminusaxis (a full two-dimensional comparison produces

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 31: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 31

00 02 04 06 08 1000

02

04

06

08

10Density

00 02 04 06 08 1000

02

04

06

08

10Pressure

00 02 04 06 08 1000

02

04

06

08

10Magnetic pressure

00 02 04 06 08 1000

02

04

06

08

10Lorentz factorMagnetic fieldlines

020 273 546 820 005 128 256 388

000 081 162 243 100 126 152 179

x

y

x

y

x

y

x

y

Figure 7 Evolved state of the magnetic rotor test problem with parameters based onthe test presented in [101] At t = 04 we show the following quantities top panelsDensity ρ and pressure P bottom panels magnetic pressure Pm equiv b2

2 and Lorentzfactor W along with numerically determined magnetic field lines Our results are inclose agreement with those shown in [67 69 101]

the same result) We note that while the overall error magnitude appears larger thanthat shown for one-dimensional and two-dimensional waves in similar works eg [106]our results are shown after five periods and we find comparable accuracy to [106] forthe first cycle

56 Loop advection

In this test originally proposed in [107] and presented in a slightly modified formby [64 106 108] a region with a circular cross section is given a non-zero azimuthalmagnetic field of constant magnitude (the ldquofield looprdquo) The magnetic field is initialised

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 32: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 32

01234567

(a) Magnetic rotor

ρ

(b)

10minus3

10minus2

10minus1

100

(c)

P

(d)

10minus3

10minus2

10minus1

100

(e)

b2 2

(f)

00 02 04 06 0810

11

12

13 (g)

W

x00 02 04 06 08

(h)

y

N = 250N = 400N = 500

Figure 8 Evolved state of the magnetic rotor test problem after t = 04 We displaywe display data for three different resolutions using 250 400 and 500 cells each to coverthe domain The data was extracted along the x = 05 and y = 05 axis respectivelyOur results are in very good agreement with figure 9 of [69]

to zero outside this region The entire fluid configuration is given a uniform velocityand periodic boundary conditions are imposed The analytical solution is for the fieldloop to move with constant velocity across the grid with no magnetic field evolutionWith rc equiv

radicx2 + y2 as the cylindrical radius we have an initial magnetic field

Bx By =minusAloopyrc Aloopxrc r lt Rloop

0 r ge Rloop (102)

In practice this configuration results from the choice of the vector potential ~A =(0 0max[0 Aloop(Rloop minus rc)]) though we note that we here set the B-field valuesexplicitly in the initial data rather than by finite differencing of the A-field an optionalso included in the code

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 33: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 33

100 200 300 400 500

10minus2

10minus1

Alfven wave test

Number of points

||Bi t fminus

Bi t i|| 2

1D2D2nd order convergence

Figure 9 Shown is the L2 norm of the difference between the evolved solution and theinitial data of the Alfveacuten wave test Convergence is observed for the one-dimensional(1D blue dotted line) and two-dimensional (2D red dashed line) test along withscaling expected for second-order convergence (solid line) In each case as a functionof Nx the number of points in the x-direction we plot the L2 norm of the B-fieldalong a line drawn in the x-direction through the centre of the computational domainmeasuring the absolute difference between the initial and final values of Bi after 5 fullwave cycles for both the one-dimensional and two-dimensional cases The resolutionsused here were (16 32 64 128) and (20times 15 40times 30 80times 60 160times 120) points in 1Dand 2D respectively

Here following [106] we choose parameters Aloop = 10minus3 ρ = 10 P = 30 forthe initial magnetic field and hydrodynamic configuration We consider two differentmodels for the direction of the field loop a ldquotwo-dimensionalrdquo case in which it is orientedvertically and a ldquothree-dimensionalrdquo case for which we rotate the direction of the tubeso that it is oriented along the xminusz face diagonal of our rectangular grid For the initialvelocity field of the two-dimensional configurations we set vx = 12 vy = 124 andconsider two different cases for the vertical velocity field parallel to the loop one in whichvz = 0 so that the motion is perpendicular to the field loop and one in which vz = 124which additionally tests the ability of the conservative-to-primitive variable inversionscheme to maintain this quantity unchanged over time For the three-dimensional casewe set vx = 02

radic2 vy = 02 vz = 01 and then rotate the tube and all vectors by

π4 radians in the x minus z-plane so that the loop runs along the x minus z face diagonalIn all cases velocity components are chosen to yield integer intervals over which theflux tube should propagate across the grid T = 24 for the two-dimensional cases andT = 5 for the three-dimensional case Runs were performed using RK2 timestepping

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 34: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 34

TVD reconstruction with an monotonized central limiter the HLLE Riemann solverdivergence cleaning active with κ = 10 and CFL factors of 04 for the 2-d cases and025 for the 3-d case

In the top panels of figure 10 we show the initial state of the flux tube of thetwo-dimensional case with Bx on the left and the magnetic pressure Pm = b22 on theright In the centre panels we show the results of a low-resolution simulation with∆x = 1128 and in the bottom panels a high-resolution simulation with ∆x = 1256Both simulations are evolved for one grid crossing time T = 24 with vz = 124 We findno substantial differences in the results after one period between vz = 0 and vz = 124While the observable decrease in the central magnetic pressure is the inevitable result ofthe simulation we find the effect is significantly decreased by the increase in resolutionWe find similar results for the three-dimensional loop whose evolution is depicted byfigure 11 and demonstrates comparable uniformity in the pressure and field strengthfor a given resolution From top to bottom we show the initial state and the statesafter one and two periods (T = 5 and T = 10 respectively) The initial pressure whichshould remain invariant is maintained throughout the body of the loop but decreasesin the central region and near the edges as expected (this result can be compared withfigure 4 of [106] with which we find excellent agreement)

57 Bondi Accretion

Bondi accretion is a steady-state solution for spherically-symmetric adiabatic accretiononto a black hole The Einstein Toolkit implementation of the general relativisticanalogue of Bondirsquos original solution [109] follows the methodology laid out in [110]assuming a fixed background spacetime of a Schwarzschild black hole of mass MBHThe solution is determined by the continuity equation

M = 4πρur2 (103)

and the integrated Euler equation for a polytropic equation of state

Q = h2(1minus 2MBH

r+ u2) (104)

where ˆ denotes a variable in Schwarzschild coordinates h is the specific enthalpyu = minusuiri and cs the speed of sound Adding a radial magnetic field to the systemdoes not alter the Bondi solution Following [14] we set the radial magnetic field byprescribing a primitive magnetic field vector of

Bi = B0M2BHx

i

radicγr3 (105)

where r =radicx2 + y2 + z2 is the coordinate radius and B0 controls the magnetic field

strength A magnetic field vector of this form by construction satisfies the divergencefree constraint (19)

A particular solution is chosen by specifying the BH mass MBH the mass accretionrate M the location of the sonic point in Schwarzschild coordinates (ie the areal

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 35: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 35

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 10 Results of the two-dimensional advected loop test with vz = 124In the top panels we show the initial state of Bx in greyscale along with the(uniform) magnetic pressure In the centre and bottom rows we show the resultof low (∆x = 1128) and high (∆x = 1256) resolution simulations after one periodhas passed (T = 24) The loss of magnetic pressure in the centre of the loop is greatlyreduced in the higher-resolution simulation

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 36: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 36

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

-05 00 05-05

00

05Bx

-05 00 05-05

00

05Magnetic pressure

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

minus10minus3minus4 middot 10minus4 4 middot 10minus4 10minus3 01 0333 0667 1 middot 10minus6

x

y

x

y

x

y

x

y

x

y

x

y

Figure 11 Results of the three-dimensional advected loop test with vz 6= 0 In thetop panels we show the 2d z = 0 slices through the initial state of Bx in greyscalealong with the (uniform) magnetic pressure In the centre and bottom rows we showthe results after one (T = 5) and two (T = 10) periods respectively The flux tubeformed by the moving flux loop is oriented diagonally in the domain hence the sliceis elliptical rather than circular The grid spacing used is ∆x = 1128 Our result isextremely similar in how well it preserves the magnetic flux and pressure of the loopto the results shown in figure 4 of [106]

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 37: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 37

radius) rs the magnetic field strength B0 and the adiabatic index Γ of the equation ofstate Table 2 lists the values of the parameters used in our simulations We explorevarious values for B0 and M but keep the sonic radius r = 8MBH and adiabatic indexΓ = 43 fixed

By requiring the solution to remain smooth at the sonic point the sound speedand thus velocity of the fluid at that point must satisfy us = cs = MBH(2rs) Fromthis equation we find the polytropic constant K for the EOS and subsequently from thecontinuity equation the conserved quantity Q The solution in the remaining domainis then determined by integrating outwards (or inwards) from the sonic point beforeconverting to the desired coordinate system of the evolution We choose to work iningoing Kerr-Schild coordinates which are horizon penetrating and therefore allow us tomodel the flow of matter across the event horizon

To avoid numerical issues due to the singularity inside of the event horizon we usea transition function

χ(r) =

12

[1 + tanh

(tan

(πr

MBHminus π

2

))]r lt MBH

1 otherwise(106)

to damp magnetic field Bi rarr χ~Bi and velocity vi rarr χvi to zero near the singularitySimilarly we reduce the mass density pressure and internal energy density to valuescorresponding to our atmosphere level ρatmo which is 10minus8 of the density at the eventhorizon

Our numerical setup follows [43] ie we simulate a rectangular box of size[0 11] times [0 11] times [minus11 11] using resolution of 011 for the low resolution and 0073for the high resolution run Since the system is spherical we used a 90 degree rotationalsymmetry around the z axis to reduce the computational costs We use a HLLE Riemannsolver TVD reconstruction using a minmod limiter We integrate in time using athird-order Runge-Kutta (RK3) integrator with Courant factor 025 For this test weemployed our constrained transport scheme which we found to be more stable in thestrongly magnetically dominated regimes explored by us in this test After each RKsubstep we re-set the grid inside of r lt 1 lt MBH and at the outer boundary to theanalytic initial solution providing Dirichlet type boundary data

We note that due to our choices of numerical timestepping method stencil sizeand resolution grid points outside of the event horizon r = 2MBH are not affected bygrid points inside of r = 1MBH Thus our method is effectively excising the interior ofr lt 1MBH from the simulated region

To verify the accuracy of our code we evolve the analytical initial data forT = 100MBH and compare the data along the z = 0055 y = 0 line to the analyticinitial data We display the result of this comparison in figure 12 We observe 2nd

order convergence with increasing resolution in the region outside of the event horizonat r = 2MBH Our results compare very with those of [43] figure 6

To adequately test the stability of the steady-state solution we repeat this testand vary both the accretion rate and the magnetic field strength We also include a

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 38: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 38

Table 2 List of tests performed for the magnetised Bondi infall The first columnsshows the run name the second the magnetic field strength parameter the third the(unitless) mass accretion rate the fourth the ration of magnetic pressure to gas densityat the horizon the fifth the resolution and the last column lists the maximum relativeerror in ρ outside of r = 2M for each test run

Run B0 M b2ρ ∆x |(ρρexact)minus 1|infinA 00 1257 00 011 970 middot 10minus5

B 00 1257 00 0073 460 middot 10minus5

C 1139 1257 250 011 742 middot 10minus2

D 1139 1257 250 0073 392 middot 10minus2

E 1139 2513 125 011 388 middot 10minus2

F 57 1257 626 011 221 middot 10minus2

G 57 2513 313 011 137 middot 10minus2

pure hydrodynamic simulation for each accretion rate In Table 2 we list the set ofparameters and their values that we vary for this test along the supremum norm of therelative error on the grid outside of the event horizon after t = 100MBH of evolution

We found that higher order reconstruction method such as PPM and methodsemploying divergence cleaning performed less well in this test In the case of divergencecleaning we suspect that our inability to specify initial data with vanishing numericaldivergence is partially the source of these difficulties Even in the constrained transportcase we observe a certain amount of initial drift away from the analytic solution in thefield values Eventually the solution settles down and we the amount drift to decreasewith increasing resolution In the case of divergence cleaning our imperfect boundarydata serves as a constant source for constraint violations possibly spoiling the resultSimilarly the interpolation inherent in adaptive mesh refinement techniques seems toserve as a source of constraint violation and prevent us from simulating this systemusing AMR and either divergence cleaning or constraint transport

58 Magnetised TOV Star

In this section we consider the stability of a stationary poloidally magnetised sphericallysymmetric neutron star model obtained via the solution of the Tolman-Oppenheimer-Volkhoff equation [111] We follow [17] and set up a poloidal magnetic field on top ofthe independently solved fluid configuration This setup provides a more challengingtest for the code than the previous ones since it probes both the GR and the MHDevolution While the initial configuration is stationary we evolve both the spacetimeand the GRMHD equations to test the ability of the new code to maintain such asolution The hydrodynamic initial data is a TOV star generated using a polytropicEOS with polytropic constant K = 100 adiabatic index Γ = 20 and initial centraldensity of 128 middot 10minus3M The fluid is evolved with a Γ-law EOS and Γ = 2 Asin [12] we introduce a poloidal field that we add on top of the TOV initial dataWe choose the vector potential Aϕ = Ab$

2(1 minus ρ0ρmax0 )np max (P minus Pcut 0) where

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 39: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 39

0 2 4 6 8 10000

015

030

Bondi accretionr0

r1 times152

2 3 4 5 6000

002

004

006

008

(a)

(b)

x [MBH]

|(ρ

ρex

act)minus

1|

Figure 12 Bondi inflow after 100MBH of evolution compared with the semi-analyticsolution for varying numerical resolution We observe 2nd order convergence in theregion outside of the event horizon at r = 2MBH The red dashed line shows theerrors in ρ compared to the analytical solution for the low resolution run r0 withresolution 011 while the black solid line shows the error in the high resolution runr1 using a resolution of 0073 scaled so that for second order convergence the curvescoincide with each other This inset shows a zoom in of the region around the eventhorizon where matter is moving fastest and the magnetic field is strongest ie wherewe expect the largest errors Our errors are almost identical to those found by [43]

$ equivradic

(xminus x)2 + y2 Here Ab determines the strength of the initial magnetic field npthe offset of the maximum of the magnetic field strength in respect to the maximumdensity ρmax of the fluid and Pcut the pressure outside of which the magnetic field is setto zero In this test we confine the magnetic field to the interior of the star by settingPcut = 1012Patmo much higher than the atmospheric pressure The parameter Ab isset to unity which together with np = 0 provides a magnetic field with a maximumstrength of 85times 10minus6 in geometrised units which corresponds to 25times 1015 G

We place the TOV star with radius 813M and gravitational mass 140M at theorigin of a grid with 5 levels of mesh refinement with cubical extents 320M 120M60M 30M and 15M The extent of the finest level is chosen in order to cover theentire star We perform three simulations with coarsest resolutions of 8M (r0) 4M

(r1) and 2M (r2) The corresponding resolutions of the finest grids are 10M (r0)05M (r1) and 025M (r2) We simulate for 69 ms using oPPM reconstruction theHLLE Riemann solver and evolve the magnetic field with constrained transport Weuse 4th-order Runge-Kutta time integration with a Courant factor of 025 We set theΓ-Driver gauge condition [59] parameter to η = 05 and add Kreiss-Oliger dissipationto the spacetime variables with a dissipation coefficient of ε = 01

Figure 13 shows the fractional change of the maximum rest-mass density ρ for

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 40: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 40

0 1 2 3 4 5 610minus5

10minus4

r0r1r2

10minus6

10minus5

r0r1times 20r2times 40

10minus3

10minus2

10minus1

|r0minus r1| |r1minus r2| times 20

093

096

099

TOV

t [ms]

ρc(

t)

ρc(

t=

0)H

nablaiB

i (|~ B|

∆x)

ε

r0r1r2

Figure 13 Results of the magnetised TOV simulation First panel Normalisedmaximum rest-mass density ρmaxρmax0 Second panel L2-norm of the Hamiltonianconstraint ||H||2 for the three resolutions r0 r1 and r2 Third panel Absolutedifference ε in normalised maximum rest-mass density between low (r0) and medium(r1) and medium (r1) and high (r2) resolutions are shown (centre panel) Fourthpanel L2-norm of the normalised divergence of the magnetic field |nablaiBi(|B| middot∆xminus1)|2 where ∆x is the finest grid spacing for each of the the three resolutions r0 r1and r2 and | ~B|infin is the maximum magnetic field strength on the grid The differencesand the L2-norm of the Hamiltonian constraint and the divergence of magnetic fieldare scaled for first order convergence

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 41: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 41

all three resolutions the differences between low (r0) and medium (r1) and medium(r1) and high (r2) resolutions in maximum rest mass density and the L2-norm of theHamiltonian constraint ||H||2 The differences in rest-mass density and the Hamiltonianconstraint are scaled for first order convergence While the TOV solution is stationarynumerical perturbations excite oscillations in the central density of the star commonlyobserved in simulations of TOV stars which must converge away with increasingresolution We can analyse the convergence properties of our simulations more carefullyby considering the second panel of figure 13 which shows the absolute differencesbetween low (r0) and medium (r1) and medium (r1) and high (r2) resolutions in thenormalised maximum rest-mass density The differences are scaled by a factor of 2 forfirst-order convergence We observe between first and second order convergence whichis the expected behaviour since the oPPM scheme reduces to first-order accuracy atthe surface of the star The third panel of figure 13 shows the L2-norm of the evolutionof the Hamiltonian constraint ||H||2 for the resolutions r0 r1 and r2 The mediumand high resolutions are rescaled for first-order convergence We again observe betweenfirst-order and second-order convergence consistent with what is expected The fourthpanel of figure 13 shows the L2-norm of the normalised divergence of the magnetic fieldas a function of time for the resolutions r0 r1 and r2 Again we observe between first-order and second-order convergence Note that the lines are constant since constrainedtransport preserves the initial constraint violation to round-off

59 Rotating Core Collapse

As a final test we present results for the collapse of a rotating magnetised stellar corewith simplified microphysics We analyse a number of quantities to demonstrate theability of our code to simulate magnetised stellar collapse in full GR For this weconsider model A1 of Shibata et al [19] who carried out simulations in axisymmetryWhile our code is 3D we can limit non-axisymmetric dynamics to even ` and m (interms of spherical harmonics) that are multiples of 4 by evolving only in one octantof the 3D domain and enforcing rotational symmetry on the two inner faces of theoctant and reflection symmetry in the z-direction We model the precollapse star as aΓ = 43-polytrope with a central density of ρc = 1010 g cmminus3 and a polytropic constant ofK = 4897middot1014[cgs] This corresponds to an ultra-relativistic degenerate electron gas atelectron fraction Ye = 05 We choose the initial rotation profile as utuφ = ϕ2

d (Ωc minus Ω)where Ωc is the central angular velocity Ω is the angular velocity and ϕd is a constantdescribing the degree of differential rotation Model A1 of [19] is uniformly rotating(ϕd rarr infin) and has a ratio of polar to equatorial radius RpRe = 0667 The initialdata were generated using the RNS code [112] During evolution we employ a hybridEOS [113 114] consisting of a two-piece polytropic cold pressure component PP and athermal pressure component Pth

P = PP + Pth (107)

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 42: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 42

0 5 10 15 20 25 30 3510minus5

10minus4

r0 r1times 125 r2times 15

10minus10

10minus9

10minus8

10minus7

r0 r1times 125 r2times 15

4

5

6

r0r1

r2Shibata et al

minus6

minus3

0

3

Rotating collapse

tminus tbounce [ms]

A2

[m]

ρc

[1014

gcmminus

3 ]H

nabla iB

i (|B|

∆x)

r0r1

r2Shibata et al

Figure 14 First panel Convergence of the gravitational wave signal A2 computedvia the quadrupole formula as in [19] and gravitational wave signal from [19] for thecorresponding model Second panel Central rest-mass density ρc Third panel L2-norm of the Hamiltonian constraint ||H||2 Fourth panel L2-norm of the normaliseddivergence of the magnetic field |nablaiBi(|B| middot ∆xminus1)|2 All four quantities are shownfor the three resolutions r0 r1 and r2 The L2-norm of the Hamiltonian constraintand the normalised divergence of the magnetic field in panels three and four are scaledfor first order convergence The constraints initially exhibit second-order convergencewhich drops to first order after core bounce generates a shock The divergence ofthe magnetic field is kept constant by the constrained transport scheme unless achange in the grid structure occurs This causes the jumps in the time evolution of|nablaiBi(|B| middot∆xminus1)|2 since the newly created grids are populated using non-constraint-preserving operators

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 43: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 43

0 5 10 15 20 25 30 351014

1015

1016

1014

1015

1016

r0r1r2

1014

1015

1016

Rotating collapse

tminus tbounce [ms]

Bz m

ax[G

]B

y max

[G]

Bx m

ax[G

]

Figure 15 Convergence of the maximum value of the magnetic field components Bifor the three different resolutions r0 r1 and r2 in the rotating core collapse simulationDifferences between the three resolutions are most obvious for Bx (top panel) but alsopresent for By (middle panel) and Bz (bottom panel) This is however to be expectedas differences in the maximum of the magnetic field components may arise from localturbulent behaviour and depend sensitively on resolution We observe a similar overallevolution of the magnetic field maxima as in [19]

Following [19] in the cold part the adiabatic index jumps from Γ1 = 13 to Γ2 = 25at ρ ge 2times 1014 g cmminus3 to mimic the stiffening of the nuclear EOS at core bounce Thethermal component of the hybrid EOS is a Γ-law with Γth = Γ1 The specific internalenergy is given by ε = εP + εth and consists of a polytropic and a thermal part Weinduce collapse by following the prescription outlined in [19] that is we set the internalenergy according to ε = 3Kρ13 and then determine the pressure as P = 3K(Γ1minus1)ρ43This reduces the pressure from its initial value by a factor of (1 minus 3(Γ1 minus 1)) (ie 10for Γ1 = 13) Following [19] we assume a poloidal initial magnetic field computed fromthe vector potential

Aφ = Ab$2 max

[ρ1nb = ρ

1nbcut 0

] (108)

with nb = 1 ρcut = 10minus4ρc and $ =radicx2 + y2 The parameter Ab is chosen so that the

maximum initial magnetic field strength is 72times 1012G For convenience we summarisethe model parameters in Table 3 Our numerical setup initially consists of 4 levels of

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 44: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 44

Table 3 Initial parameters and properties of model A1

Central rest-mass density [g cmminus3] ρc 10times 1010

ADM mass [M] MADM 1503Baryonic mass [M] MB 1503Ratio of rotational kinetic to potential energy TW 89times 10minus3

Equatorial radius [km] Re 2267non-dimensional angular momentum parameter JM2

ADM 1235Ratio of polar to equatorial radius zRe 0667Differential rotation parameter A infinAngular velocity at the rotation axis Ωc 411Maximum initial magnetic field strength [G] |Bi|max 72times 1012

Maximum initial ratio of magnetic and fluid pressure (PmagP )max 21times 10minus4

Ratio of magnetic to rotational kinetic energy EmagTrot 19times 10minus3

nested grids each refined by a factor of 2 in resolution As the star collapses and thecentral density increases we progressively add 5 additional refinement levels followingthe approach taken in [115] The density thresholds are chosen to be [80times1010 32times1011128times1012 512times1012 205times1013] g cmminus3 The final grid structure consists of 9 levels ofnested grids with cubical extents [56698 km 30617 km 2438 km 15592 km 2835 km2126 km 1447 km 591 km 236 km] The extent of the finest level is chosen such thatit covers the entire compact remnant (the ldquoproto-NSrdquo) after core bounce The inner-most refinement level for our fiducial setup (r1) has a linear resolution of 3691 m Weconsider two more resolutions r0 with fine grid resolution 4614 m and r2 with fine gridresolution 2953 m This allows us to study the resolution dependence of our results Weuse vertex-centred AMR constrained transport and TVD reconstruction with the vanLeer monotonized central limiter While GRHyro includes higher-order reconstructionschemes we use TVD in these simulations to match the numerical methods used in [19]as closely as possible The time integration is performed using RK4 with a Courantfactor of 02 The damping coefficient of the Γ-driver gauge condition for the spacetimeevolution is set to η = 12 We add 5th-order Kreiss-Oliger dissipation to the spacetimevariables with a dissipation coefficient ε = 01 The atmosphere is chosen to be a factor1010 smaller than the central density Shibata et al [19] extract the gravitational wavesignal from rotating core collapse and bounce via a quadrupole formula and define awave amplitude

A2 = Ixx minus Izz (109)

with the definitions of Ixx and Ixx given in [19] Since the gravitational wave signal isa good tracer of the overall dynamics of rotating core collapse we extract A2 from oursimulations for comparison with [19]

In figure 14 we show convergence for the central density ρc the gravitationalwave signal A2 the L2-norm of the Hamiltonian constraint ||H||2 and the normaliseddivergence of the magnetic field |nablaiB

i(|B| middot ∆xminus1)|2 The Hamiltonian constraint inthe third panel of figure 14 and the normalised divergence of the magnetic field in the

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 45: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 45

fourth panel are scaled for first-order convergence Before core bounce second orderconvergence is achieved but reduced to first order after bounce Our numerical schemereduces to first-order accuracy at shocks contact discontinuities and local maxima ieat the centre of the proto-NS which dominates the Hamiltonian constraint violationThe gravitational wave signal A2 the central rest-mass density ρc the L2-norm of theHamiltonian constraint and the normalised divergence of the magnetic field are shownfor the three resolutions r0 r1 and r2 In the top two panels we also show thegravitational wave signal and central density evolution as extracted and digitised fromfigures 2 and 17 of [19] We observe excellent agreement for the gravitational waveamplitude A2 with the results of [19] The central density evolution of [19] reaches apeak value of 65 middot 1014 g cmminus3 at bounce while we find a maximum bounce value of64 middot 1014 g cmminus3 The post-bounce evolution differs by an approximately constant offsetof 6 however the features in the time evolution itself are very similar We attributethe observed deviations in maximum rest-mass density evolution to differences in ournumerical setup

We find between second-order and first-order convergence for the gravitationalwave signal A2 the central density ρc and the L2-norm of the Hamiltonian constraint||H||2 which is within the expected behaviour of the code The convergence inboth central density and maximum rest-mass evolution are less clean than for theHamiltonian constraint due to their oscillatory nature The fourth panel of figure 14shows the evolution of the L2-norm of the normalised divergence of the magnetic field|nablaiB

i(|B| middot ∆xminus1)|2 The constrained transport scheme used in this test maintainsthe divergence present in the initial data to round-off precision Constraint violationsappear only when additional grids are added and filled by non-constraint-preservinginterpolation

Figure 15 shows the maximum of the magnetic field components |Bi|max as afunction of time for the three resolutions r0 r1 and r2 We observe resolution-dependent differences in the magnetic field evolution in the postbounce phase This isparticularly obvious for Bx but also visible in Bz This behaviour is expected since theevolution of the magnetic field is very sensitive to resolution due to effects of turbulenceand instabilities most importantly the magnetorotational instability (MRI) [116 117]A similar behaviour for the resolution dependence of Bi is shown in figure 6 of [19] Ourresults for the postbounce magnetic-field evolution differ quantitatively from [19] whichis expected due to our different numerical setup While our setup consists of differentlayers of mesh refinement with varying resolution throughout our 3D domain [19] employa uniform axisymmetric grid with a coordinate singularity at the rotation axis

6 Conclusions

We have presented the implementation and a set of rigorous tests of ideal general-relativistic magnetohydrodynamics (GRMHD) in the GRHydro code which is part ofthe Einstein Toolkit All tests and results presented in this paper can be reproduced

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 46: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 46

with the code and input files available at httpeinsteintoolkitorgGRHydro is the first open-source code that provides for fully self-consistent GRMHD

evolutions on dynamical general-relativistic spacetimes Its development includesmany contributions from a wide array of collaborators involved with other relativisticcodes particularly the original version of the general-relativistic hydrodynamics codeWhisky [32] from which GRHydro started and the publicly available fixed-backgroundHARM code [44] from which we have adapted a number of routines and techniquesCombined with other functionality from the Einstein Toolkit the code may be usedto perform GRMHD evolutions of a broad range of relativistic astrophysical systemsWe expect these to primarily consist of magnetised isolated and binary neutron starsand collapsing stellar cores but should the proper initial data be constructed accretiondisks and tori white dwarfs and a number of other configurations could be simulatedas well

Our implementation of the equations of GRMHD also features the ability toeliminate spurious divergence in the magnetic field through two different methodshyperbolic divergence cleaning and constrained transport While these are clearlysufficient for many problems one of the responsibilities in implementing a community-based code is to incorporate techniques developed by other groups to solve similarproblems Along these lines we foresee work in the future to incorporate vectorpotential treatments in which the B-field is computed from the values of an underlyingvector potential A that serves as a fundamental evolved variable Such a scheme hasbeen implemented by Etienne et al [69 92] along with work to determine the properelectromagnetic gauge conditions to implement for compact object binaries involvingneutron stars and other systems of interest

The GRHydro code now includes several improved higher-order techniques forreconstruction of states at cell interfaces for use in high-resolution shock capturingincluding three that were described in the original Einstein Toolkit code paper [22]TVD PPM and ENO as well as three new methods that have been implemented andreleased since ePPM [36 78] WENO5 [79 86] and MP5 [80] (see Appendix B)

In the near future we will extend the new GRMHD version of GRHydro tothe generalised multipatch infrastructure of [36] which will be released as part ofthe Einstein Toolkit and will enable simulations on a combination of Cartesian andcurvilinear grids Further work will be directed towards implementing improvedapproximate Riemann solvers for GRMHD While the Roe [118] and Marquina [119 120]solvers implemented in GRHydro for purely hydrodynamic evolutions are likely tobe too computationally expensive requiring an eigenmode breakdown of an 8 times 8matrix system at every grid cell the HLLC solver which recovers the contact wavedropped by the HLLE approximation may be an attractive and computationallyefficient improvement [74 121] Additionally we will introduce improved techniques forperforming conservative-to-primitive variable conversions to handle cases in which thecurrent routines are known to fail particularly those where the Lorentz factor becomesvery large or the fluid becomes dominated by magnetic pressure [122]

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 47: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 47

In the long term we expect the GRHydro code and the Einstein Toolkit toevolve into a toolkit not just for numerical relativity but for the broad computationalrelativistic astrophysics community Towards this end we will include additional physicscapabilities in future versions of the Toolkit These will include better treatments offinite-temperature and composition dependence nuclear reaction networks radiationtransport of neutrinos and photons and improvements over the current ideal GRMHDapproximation following the most recent developments in resistive GRMHD [123 124]

Acknowledgments

We thank Bruno Giacomazzo for use of his exact Riemann solver code as well ashelpful conversations on a number of topics related to the work here We also thankE Abdikamalov M Campanelli M Duez Z Etienne B Farris U Gamma P LagunaL Lehner C Lousto C Palenzuela V Paschilidis J Penner and Y Zlochower foruseful conversations related to various aspects of the presented work

The Einstein Toolkit is directly supported by the National Science Foundation inthe USA under the grant numbers 0903973 0903782 0904015 (CIGR) and 1212401 1212426 1212433 1212460 (Einstein Toolkit) Related grants contribute directly andindirectly to work in support of the Einstein Toolkit including NSF AST-1028087NSF PHY-1214449 NSF AST-1212170 CAREER NSF PHY-0969855 NSF OCI-0905046 DMS-0820923 NASA 08-ATFP08-0093 an NSERC grant to Erik Schnetterand Deutsche Forschungsgemeinschaft grant SFBTransregio 7 ldquoGravitational-WaveAstronomyrdquo Christian Ott acknowledges support from the Alfred P Sloan FoundationRoland Haas acknowledges support by the Natural Sciences and Engineering Councilof Canada Christian Reisswig acknowledges support by NASA through EinsteinPostdoctoral Fellowship grant number PF2-130099 awarded by the Chandra X-raycenter which is operated by the Smithsonian Astrophysical Observatory for NASAunder contract NAS8-03060 Tanja Bode acknowledges support from NSF grants094141712058564

Portions of this research were conducted with computational resources provided byLouisiana State University (allocations hpc_cactus hpc_numrel and hpc_hyrel) by theLouisiana Optical Network Initiative (allocations loni_cactus and loni_numrel) by theNational Science Foundation through XSEDE resources (allocations TG-PHY060027NTG-ASC120003 TG-PHY100033 TG-MCA02N014 and TG-PHY120016) the GeorgiaTech FoRCE cluster and the Caltech compute cluster Zwicky (NSF MRI award PHY-0960291)

The multi-dimensional visualizations were generated with the open-source VisItvisualisation package [50 125] All other figures were generated with the Python-basedmatplotlib package [126]

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 48: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 48

Table A1 Overview of differences in variable notation between HARM and ValenciaQuantity ValenciaET HARM

Pressure P p

Energy density ρh equiv ρ(1 + ε) + P w

3-velocity for a normal observer vi equiv ui

γ+ βi

αvi

Lorentz factor W γ

Projected velocity Wvi ui equiv jimicroumicro = ui + Wβi

αMomentum density for a normalobserver

Smicro equiv αT 0micro Qmicro

Projected momentum density Smicro equiv (δνmicro + nmicronν)Sν Qmicro

Auxiliary Con2prim variable Q equiv ρhW 2 W

Appendix A Notation in HARM and Valencia formulations

The conservative to primitive inversion routine found in the Einstein Toolkit wasoriginally developed as part of the HARM GRMHD code [63] and reflects its notationinternally For historical reasons the development of GRMHD in the Einstein Toolkitfollows the Valencia formulation of equations [6 37ndash39 73] and this notation was usedthroughout the first paper summarising the Einstein Toolkit [22] To convert from oneset of notation to the other we note that the equivalence of various physical quantitiesin Table A1

Appendix B Comparison of reconstruction methods

We briefly compare the new reconstruction methods (section 42) that have beenimplemented in GRHydro In [36] a detailed comparison between ePPM and oPPMfor cell-centred and vertex-centred AMR was performed These authors foundthat ePPM is superior to oPPM in all cases they considered and especially inthose using cell-centred AMR Here we compare oPPM ePPM WENO5 and MP5reconstruction using the example of an magnetised dynamically evolved TOV starusing vertex-centred AMR We use 7 levels of mesh refinement with cubical extent[400M 200M 50M 30M 19M 12M] and employ octant symmetry The finestlevel has a resolution of ∆x = 02M and contains the entire star We use the HLLERiemann solver We evolve spacetime with a Γ-driver gauge parameter of η = 05and 5th-order Kreiss-Oliger dissipation with ε = 01 is added to the right-hand side ofspacetime variable We integrate in time with a fourth-order Runge-Kutta integratorand a Courant factor of 04 We construct the initial initial data by solving the TOVequations using a polytropic EOS with Γ = 2 and scale parameter K = 100M Thecentral density is set to ρc = 128 times 10minus3Mminus2

During the evolution we use a Γ-lawEOS with Γ = 2

In figure B1 we show the L2-norm of the Hamiltonian constraint ||H||2 as a function

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 49: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 49

00 04 08 12 16 20 24 28

015

030

045

TOV reconstruction test

H 2

[10minus

6 ]

t [ms]

oPPMePPM

WENO5MP5

Figure B1 Comparison of oPPM ePPM WENO5 and MP5 reconstruction basedon the example of a dynamically evolved TOV star We show the L2-norm of theHamiltonian constraint ||H||2 MP5 performs best while oPPM performs worst inthis simple test

of time All simulation settings are identical except for the reconstruction method Wefind that MP5 reconstruction performs better than ePPM or WENO5 in this simpletest WENO5 performs worse than ePPM and MP5 The worst results are obtainedwith oPPM which reduces to first order at the smooth density maximum at the centerof the star In the vertex-centred case however the error due to oPPM reconstructionis not as large as in the cell-centred case [36] since in the former the central densitymaximum is approximately located on a grid point With oPPM the dominant sourceof error is thus generated at the center of the star The other reconstruction methodsmaintain high order at the smooth density maximum With them the dominant sourceof error is generated at the steep density drop at the surface of the star We emphasisehowever that this test is only one of many possible tests A more detailed comparisonwould be necessary for more general conclusions

Appendix C Full set of planar MHD shocktubes

The five test cases collected in [96] are widely used to test new MHD codes We verifiedthat our code passes all tests and compare the results to the exact solution of [43] Themain text 52 contains detailed description of our test setup

In figures C1 and C2 we present results for the moderate and strong blast wave testsof [96] which prove much more challenging to the code due to the Lorentz contractionof the fast moving shock Finally in figure C3 we present the non-coplanar test problemof [96]

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 50: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 50

051015202530

(a)

ρ

Exact solution1D numerical2D numerical

0

10

20

30

(b)

P

0

02

04

06(c)

vx

minus02

minus01

0(d)

vy

1011121314

(e)

W

minus04 minus02 00 02 04

2

4

6(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 2 Test

Figure C1 Evolution of the Balsara2 shocktube case performed with divergencecleaning with all conventions as in figure 3 The numerical code reproduces theanalytical result very well with only some oscillations visible in the density ρ nearthe shock front capturing the sharp shock features at least as well as [96] A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 51: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 51

0

5

10

15

(a)

ρ

Exact solution1D numerical2D numerical

0200400600800

1000(b)

P

002040608

1(c)

vx

minus02

minus01

0 (d)

vy

1

2

3 (e)

W

minus04 minus02 00 02 04

2

4

6 (f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 3 Test

Figure C2 Evolution of the Balsara3 shocktube case performed with divergencecleaning with all conventions as in figure 3 This test proved more demanding onour code in particular the narrow right moving fast shock is not resolved well using1600 grid points resulting in a sizeable overshoot near the shock This feature iscommon to other codes [96] independent of the specific formulation used A moredetailed description of the test setup and parameters can be found in the main text insection 52

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 52: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

GRHydro A new open source GRMHD code for the Einstein Toolkit 52

10

15

20

25

(a)

ρ

1

2

3

4(b)

P

minus040minus020

00204

(c)

vx

minus02minus01

0010203

(d)

vy

Exact solution1D numerical2D numerical

100105110115120

(e)

W

minus04 minus02 00 02 04minus15

minus10

minus05

00

05

(f)

By

minus04 minus02 00 02 04

minus005

000

005 (g)

nablamiddotB

x

Balsara 5 Test

Figure C3 Evolution of the Balsara5 shocktube case performed with divergencecleaning with all conventions as in figure 3 A more detailed description of the testsetup and parameters can be found in the main text in section 52

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 53: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

REFERENCES 53

Appendix D Monopole tests modelling high frequency noise

We also include initial data in our tests that models high-frequency numerical noise inthe magnetic field Here the spatial dependence of Bx is set as in section 51 and theother components of the B-field are set to the same amplitude or to zero with non-zero components multiplied by (minus1)N(ijk) where N(i j k) is a function of the indicesi j and k of the grid cell in the x y and z-directions to produce an alternatingpattern For a one-dimensional alternating pattern we set N(i j k) = i + j + k andonly set Bx to be non-zero and alternating For a two-dimensional test we use the samefunction for N(i j k) and apply it to Bx to make it alternate but set By to the originalGaussian state and Bz = 0 For a three-dimensional test we set N(i j k) = i + j sothe alternation is two-dimensional in character but set Bx = By = Bz and allow allthree to alternate

Figure D1 shows the performance of the divergence cleaning algorithm in removinghigh frequency constraint violating noise such as noise generated during the numericalevolution Clearly the algorithm is very efficient in damping out the high frequencyconstraint violations reducing them much more quickly than the simple Gaussian pulsepresent in figure 2

References

[1] Shibata M and Uryu K 2000 Phys Rev D 61 064001[2] Pretorius F 2005 Phys Rev Lett 95 121101[3] Baker J G Centrella J Choi D I Koppitz M and van Meter J 2006 Phys Rev Lett 96 111102[4] Campanelli M Lousto C O Marronetti P and Zlochower Y 2006 Phys Rev Lett 96 111101[5] Ott C D Reisswig C Schnetter E OrsquoConnor E Sperhake U Loumlffler F Diener P Abdikamalov

E Hawke I and Burrows A 2011 Phys Rev Lett 106 161103[6] Font J A 2008 Living Rev Relativity 11 7 URL (cited on 01ndash01ndash2012) httpwww

livingreviewsorglrr--2008

[7] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1[8] Hawley J F 2009 Astrophys Space Sci 320 107[9] Anderson M Hirschmann E W Lehner L Liebling S L Motl P M Neilsen D Palenzuela C and

Tohline J E 2008 Phys Rev Lett 100 191101[10] Giacomazzo B Rezzolla L and Baiotti L 2009 Mon Not Roy Astron Soc 399 L164ndashL168[11] Giacomazzo B Rezzolla L and Baiotti L 2011 Phys Rev D 83 044014[12] Liu Y T Shapiro S L Etienne Z B and Taniguchi K 2008 Phys Rev D 78 024012[13] Chawla S Anderson M Besselman M Lehner L Liebling S L Motl P M and Neilsen D 2010

Phys Rev Lett 105 111101[14] Etienne Z B Liu Y T Paschalidis V and Shapiro S L 2012 Phys Rev D 85 064029[15] Shibata M Sekiguchi Y and Takahashi R 2007 Prog Theor Phys 118 257[16] Shibata M and Sekiguchi Y 2012 Prog Theor Phys 127 535[17] Duez M D Liu Y T Shapiro S L Shibata M and Stephens B C 2006 Phys Rev Lett 96 031101[18] Lehner L Palenzuela C Liebling S L Thompson C and Hanna C 2012 Phys Rev D86 104035

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 54: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

REFERENCES 54

minus2 minus1 0 1 2

minus4

minus2

0

2

4

nablamiddotBtimes

103

x

κ = 100κ = 10κ = 1

κ = 100κ = 10κ = 1

t = 00 t = 025 t = 05

t = 075 t = 10 t = 125

t = 15 t = 175 t = 20

κ = 100κ = 10κ = 1

Figure D1 Behaviour of our divergence cleaning scheme demonstrated by monopoledamping and advection for a three-dimensional alternating Gaussian configurationConventions are the same as in figure 2 We still see more significant divergencestravelling further from their initial location for κ = 1 but damping is markedly fasterfor the case κ = 100 when dealing with high-frequency noise as compared to thelower-frequency modes characterising the Gaussian monopole seemingly because ofcancellation effects from neighbouring points The choice κ = 10 still yields the bestresults for numerically eliminating spurious divergence of the magnetic field

[19] Shibata M Liu Y T Shapiro S L and Stephens B C 2006 Phys Rev D 74 104026[20] Giacomazzo B Baker J G Miller M C Reynolds C S and van Meter J R 2012 Astrophys J Lett

752 L15[21] Farris B D Gold R Paschalidis V Etienne Z B and Shapiro S L 2012 Phys Rev Lett 109

221102[22] Loumlffler F Faber J Bentivegna E Bode T Diener P Haas R Hinder I Mundim B C Ott C D

Schnetter E Allen G Campanelli M and Laguna P 2012 Class Quantum Grav 29 115001[23] Einstein Toolkit Open Software for Relativistic Astrophysics URL httpeinsteintoolkit

org

[24] Cactus Computational Toolkit URL httpwwwcactuscodeorg

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 55: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

REFERENCES 55

[25] Goodale T Allen G Lanfermann G Massoacute J Radke T Seidel E and Shalf J 2003 The Cactusframework and toolkit Design and applications Vector and Parallel Processing ndash VECPARrsquo20025th International Conference Lecture Notes in Computer Science (Berlin Springer) URLhttpedocmpgde3341

[26] Allen G Goodale T Lanfermann G Radke T Rideout D and Thornburg J 2011 Cactus UsersrsquoGuide URL httpwwwcactuscodeorgGuidesStableUsersGuideUsersGuideStablepdf

[27] Schnetter E Hawley S H and Hawke I 2004 Class Quantum Grav 21 1465[28] Schnetter E Diener P Dorband E N and Tiglio M 2006 Class Quantum Grav 23 S553[29] Carpet Adaptive Mesh Refinement for the Cactus Framework URL httpwwwcarpetcode

org

[30] Brown J D Diener P Sarbach O Schnetter E and Tiglio M 2009 Phys Rev D 79 044023[31] Reisswig C Ott C D Sperhake U and Schnetter E 2011 Phys Rev D 83 064008[32] Baiotti L Hawke I Montero P J Loumlffler F Rezzolla L Stergioulas N Font J A and Seidel E

2005 Phys Rev D 71 024035[33] Hawke I Loumlffler F and Nerozzi A 2005 Phys Rev D 71 104006[34] Baiotti L Hawke I Montero P J and Rezzolla L 2003 Mem Soc Ast It 74 S210[35] Whisky ndash relativistic hydrodynamics and magnetohydrodynamics URL httpwww

whiskycodeorg

[36] Reisswig C Haas R Ott C D Abdikamalov E Moesta P Pollney D and Schnetter E 2013Accepted for publication in Phys Rev D arXiv12121191

[37] Martiacute J M Ibaacutentildeez J M and Miralles J M 1991 Phys Rev D 43 3794[38] Banyuls F Font J A Ibanez J M Marti J M and Miralles J A 1997 Astrophys J 476 221[39] Ibaacutentildeez J Aloy M Font J Martiacute J Miralles J and Pons J 1999 Riemann solvers in general

relativistic hydrodynamics Proceedings of Godunov Methods Theory and Applications 1999 edToro E (New York Kluwer AcademicPlenum Publishers) p 485 ISBN 978-0306466014

[40] Toacuteth G 2000 J Comp Phys 161 605[41] Liebling S L Lehner L Neilsen D and Palenzuela C 2010 Phys Rev D 81 124023[42] Penner A J 2011 Mon Not Roy Astron Soc 414 1467[43] Giacomazzo B and Rezzolla L 2007 Class Quantum Grav 24 S235[44] Gammie C F McKinney J C and Toth G 2003 Astrophys J 589 444[45] Seidel E L Allen G Brandt S R Loumlffler F and Schnetter E 2010 Simplifying CoMplex Software

Assembly The Component Retrieval Language and Implementation Proceedings of the 2010TeraGrid Conference (Preprint arXiv10091342[csPL])

[46] Allen G Goodale T Loumlffler F Rideout D Schnetter E and Seidel E L 2010 ComponentSpecification in the Cactus Framework The Cactus Configuration Language Grid2010Proceedings of the 11th IEEEACM International Conference on Grid Computing (PreprintarXiv10091341[csDC])

[47] Thomas M and Schnetter E 2010 Simulation factory Taming application configuration andworkflow on high-end resources Grid Computing (GRID) 2010 11th IEEEACM InternationalConference on p 369 (Preprint arXiv10084571[csDC])

[48] SimFactory Herding numerical simulations URL httpsimfactoryorg

[49] Korobkin O Allen G Brandt S R Bentivegna E Diener P Ge J Loumlffler F Schnetter E andTao J 2011 Runtime analysis tools for parallel scientific applications Proceedings of the 2011TeraGrid Conference Extreme Digital Discovery TG rsquo11 (New York NY USA ACM) p 2218ISBN 978-1-4503-0888-5

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 56: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

REFERENCES 56

[50] Childs H Brugger E S Bonnell K S Meredith J S Miller M Whitlock B J and Max N 2005 Acontract-based system for large data visualization Proceedings of IEEE Visualization 2005 p 190

[51] Berger M J and Oliger J 1984 J Comput Phys 53 484[52] Husa S Hinder I and Lechner C 2006 Comput Phys Commun 174 983[53] Lechner C Alic D and Husa S 2004 Analele Universitatii de Vest din Timisoara Seria

Matematica-Informatica 42[54] Kranc Kranc assembles numerical code URL httpkranccodeorg

[55] Nakamura T Oohara K and Kojima Y 1987 Prog Theor Phys Suppl 90 1[56] Shibata M and Nakamura T 1995 Phys Rev D 52 5428[57] Baumgarte T W and Shapiro S L 1999 Phys Rev D 59 024007[58] Alcubierre M Bruumlgmann B Dramlitsch T Font J A Papadopoulos P Seidel E Stergioulas N

and Takahashi R 2000 Phys Rev D 62 044034[59] Alcubierre M Bruumlgmann B Diener P Koppitz M Pollney D Seidel E and Takahashi R 2003

Phys Rev D 67 084023[60] Hyman J M 1976 The method of lines solution of partial differential equations Tech Rep COO-

3077-139 ERDA Mathematics and Computing Laboratory Courant Institute of MathematicalSciences New York University

[61] Harm2d Code URL httprainmanastroillinoiseducodelibcodesharmsrc

[62] Noble S C Krolik J H and Hawley J F 2009 Astrophys J 692 411[63] Noble S C Gammie C F McKinney J C and Del Zanna L 2006 Astrophys J 641 626[64] Stone J M Gardiner T A Teuben P Hawley J F and Simon J B 2008 Astrophys J Suppl 178

137[65] Athena code URL httpstracprincetoneduAthena

[66] Del Zanna L Zanotti O Bucciantini N and Londrillo P 2007 Astron Astrophys 473 11[67] Shibata M and Sekiguchi Y i 2005 Phys Rev D 72 044014[68] Duez M D Liu Y T Shapiro S L and Stephens B C 2005 Phys Rev D 72 024028[69] Etienne Z B Liu Y T and Shapiro S L 2010 Phys Rev D 82 084031[70] Anderson M Hirschmann E Liebling S L and Neilsen D 2006 Class Quant Grav 23 6503[71] Arnowitt R L Deser S and Misner C W 2008 General Relativity and Gravitation 40 1997[72] York Jr J W 1983 The initial value problem and dynamics Gravitational Radiation ed N Deruelle

amp T Piran (North-Holland Publishing Co) pp 175ndash201[73] Anton L Zanotti O Miralles J A Marti J M Ibanez J M et al 2006 Astrophys J 637 296[74] Toro E F 1999 Riemann Solvers and Numerical Methods for Fluid Dynamics (Berlin Springer)[75] Colella P and Woodward P R 1984 J Comp Phys 54 174[76] Harten A Engquist B Osher S and Chakravarthy S R 1987 J Comp Phys 71 231[77] Shu C W 1999 High order methods for computational physics ed Barth T J and Deconinck H A

(New York Springer) pp 439ndash582[78] McCorquodale P and Colella P 2011 Comm Appl Math Comp Sci 6 1[79] Shu C W 1998 Lect Not Math 1697 325[80] Suresh A and Huynh H T 1997 J Comp Phys 136 83[81] Harten A Lax P D and van Leer B 1983 SIAM review 25 35[82] Einfeldt B 1988 SIAM J Numer Anal 25 294[83] Giacomazzo B 2007 General Relativistic Magnetohydrodynamics fundamental aspects and

applications PhD thesis SISSA Scuola Internazionale Superiore di Studi Avanzati Trieste ITALYURL httpdigitallibrarysissaithandle19631916

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 57: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

REFERENCES 57

[84] Roe P L 1986 Ann Rev Fluid Mech 18 337[85] van Leer B 1977 J Comp Phys 23 263[86] Tchekhovskoy A McKinney J C and Narayan R 2007 Mon Not Roy Astron Soc 379 469[87] Harten A 1983 J Comp Phys 49 357[88] Mignone A Tzeferacos P and Bodo G 2010 J Comput Phys 229 5896[89] Penner A J 2011 Numerical Analysis of the Ultrarelativistic and Magnetized Bondi-Hoyle Problem

PhD thesis University of British Columbia Vancouver BC Canada URL httphdlhandlenet242936734

[90] GRHydro documentation URL httpeinsteintoolkitorgdocumentationThornGuideThornGuidech152htmlx171-2681000R1

[91] Dedner A Kemm F Kroumlner D Munz C D Schnitzer T and Wesenberg M 2002 J Comp Phys175 645ndash673

[92] Etienne Z B Paschalidis V Liu Y T and Shapiro S L 2012 Phys Rev D 85 024013[93] Evans C R and Hawley J F 1988 Astrophys J 332 659ndash677[94] Yee K 1966 Antennas and Propagation IEEE Transactions on 14 302[95] Balsara D S and Spicer D S 1999 J Comp Phys 149 270[96] Balsara D 2001 Astrophys J Suppl 132 83[97] Brio M and Wu C C 1988 J Comput Phys 75 400[98] Giacomazzo B and Rezzolla L 2006 J Fluid Mech 562 223[99] Giacomazzo B and Rezzolla L An Exact Riemann Solver for Relativistic MHD URL http

wwwbrunogiacomazzoorgexactriemannsolvhtml

[100] Balsara D S 1998 Astrophys J Suppl 116 133 URL httpstacksioporg0067-0049116i=1a=133

[101] Del Zanna L Bucciantini N and Londrillo P 2003 Astron Astrophys 400 397[102] Komissarov S S 1999 Mon Not R Astron Soc 303 343[103] Kreiss H and Oliger J 1973 Methods for the Approximate Solution of Time Dependent Problems

(Global Atmospheric Research Programme (GARP) GARP Publication Series vol 10) (GARPPublication)

[104] Runge C 1895 Mathematische Annalen 46 167 URL httpdxdoiorg101007BF01446807

[105] Kutta W 1901 Z Math Phys 46 435[106] Beckwith K and Stone J M 2011 Astrophys J Suppl 193 6[107] Devore C R 1991 J Comput Phys 92 142[108] Toacuteth G and Odstrcil D 1996 J Comput Phys 128 82ndash100[109] Bondi H 1952 Mon Not Roy Astron Soc 112 195[110] Shapiro S and Teukolsky S 1983 Black holes white dwarfs and neutron stars The physics of

compact objects (New York Chichester Brisbane Toronto Singapore Wiley amp Sons)[111] Tolman R C 1939 Phys Rev 55 364ndash373[112] Stergioulas N and Friedman J L 1995 Astrophys J 444 306[113] Janka H T Zwerger T and Moumlnchmeyer R 1993 Astron Astrophys 268 360[114] Dimmelmeier H Font J A and Muumlller E 2002 Astron Astrophys 393 523[115] Ott C D Dimmelmeier H Marek A Janka H T Hawke I Zink B and Schnetter E 2007 Phys

Rev Lett 98 261101[116] Balbus S A and Hawley J F 1991 Astrophys J 376 214ndash233[117] Balbus S A and Hawley J F 1998 Rev of Mod Phys 70 1ndash53

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise
Page 58: GRHydro: Anewopensourcegeneral-relativistic ...pmoesta/grmhdcode.pdf · same open-source licensing terms as other components in the Toolkit. Given the public nature of the project,

REFERENCES 58

[118] Roe P L 1981 J Comp Phys 43 357[119] Donat R and Marquina A 1996 J Comp Phys 125 42[120] Aloy M Ibanez J Marti J and Muller E 1999 Astrophys J Suppl 122 151[121] Mignone A and Bodo G 2006 Mon Not R Astron Soc 368 1040[122] Mignone A and McKinney J C 2007 Mon Not R Astron Soc 378 1118ndash1130[123] Dionysopoulou K Alic D Palenzuela C Rezzolla L and Giacomazzo B 2012 (Preprint arXiv

12083487[gr-qc])[124] Palenzuela C 2012 (Preprint arXiv12120130[astro-phHE])[125] VisIt visualization tool URL httpswcillnlgovcodesvisit

[126] Hunter J D 2007 Comput Sci Eng 9 90

  • Introduction
  • Numerical Relativistic Hydrodynamics and GRMHD codes
    • The Einstein Toolkit
    • Other relativistic MHD codes
      • The Valencia formulation of ideal MHD
      • Numerical Methods
        • Evaluation of magnetic field expressions
        • Reconstruction
          • TVD reconstruction
          • PPM reconstruction
          • ENO reconstruction
          • WENO reconstruction
          • MP5 reconstruction
            • Riemann Solver
            • Conservative to Primitive variable transformations
              • General EOS
              • Barotropic EOS
                • Divergence-free Constraint Treatment
                  • Divergence cleaning
                  • Constrained transport
                      • Tests
                        • Monopole tests
                        • Planar MHD Shocktubes
                        • Cylindrical Shocks
                        • Magnetic rotor
                        • Alfveacuten Wave
                        • Loop advection
                        • Bondi Accretion
                        • Magnetised TOV Star
                        • Rotating Core Collapse
                          • Conclusions
                          • Notation in HARM and Valencia formulations
                          • Comparison of reconstruction methods
                          • Full set of planar MHD shocktubes
                          • Monopole tests modelling high frequency noise