Eur. Phys. J. D (2014)DOI: 10.1140/epjd/e2014-50211-7

Regular Article

THE EUROPEANPHYSICAL JOURNAL D

Gyroaverage operator for a polar mesh�

Christophe Steiner1, Michel Mehrenberger1,2,a, Nicolas Crouseilles3, Virginie Grandgirard4,Guillaume Latu4, and Fabien Rozar4,5,b

1 IRMA, Universite de Strasbourg, France2 Max-Planck-Institut fur Plasmaphysik, Garching, Germany3 Inria-Rennes Bretagne Atlantique, IPSO team et IRMAR, Universite de Rennes 1, France4 CEA/DSM/IRFM, Association Euratom-CEA, Cadarache, France5 Maison de la simulation, CEA Saclay, Gif sur Yvette, France

Received 15 March 2014 / Received in final form 1st August 2014Published online (Inserted Later) – c© EDP Sciences, Societa Italiana di Fisica, Springer-Verlag 2014

Abstract. In this work, we are concerned with numerical approximation of the gyroaverage operatorsarising in plasma physics to take into account the effects of the finite Larmor radius corrections. The workinitiated in reference [N. Crouseilles, M. Mehrenberger, H. Sellama, CiCP 8, 484 (2010)] is extended here topolar geometries. A direct method is proposed in the space configuration which consists in integrating on thegyrocircles using interpolation operator (Hermite or cubic splines). Numerical comparisons with a standardmethod based on a Pade approximation are performed: (i) with analytical solutions; (ii) considering the4D drift-kinetic model with one Larmor radius and (iii) on the classical linear DIII-D benchmark case[A. Dimits et al., Phys. Plasmas 7, 969 (2000)]. In particular, we show that in the context of a drift-kineticsimulation, the proposed method has similar computational cost as the standard method and its precisionis independent of the radius.

1 Introduction1

In strongly magnetized plasma, when collisions are not2

the dominant collective effect, one has to deal with ki-3

netic models. Fluid models, which assume that the dis-4

tribution function is close to an equilibrium, are no more5

suitable. However, the numerical solution of Vlasov type6

models is challenging since this model involves six dimen-7

sions in the phase space. Moreover, multi-scaled phenom-8

ena make the problem very difficult. Gyrokinetic theory9

enables to get rid of one of these constraints since the ex-10

plicit dependence on the phase angle of the Vlasov equa-11

tion is removed through gyrophase averaging while gyrora-12

dius effects are retained. The so-obtained five-dimensional13

function should be self-consistently coupled with Maxwell14

equations. In the following we consider the electrostatic15

approximation, where Maxwell equations are reduced to16

� Contribution to the Topical Issue “Theory and Applica-tions of the Vlasov Equation”, edited by Francesco Pegoraro,Francesco Califano, Giovanni Manfredi and Philip J. Morrison.

a e-mail: mehrenbe@math.unistra.frb A part of this work was carried out using the Helios su-

percomputer system at Computational Simulation Centre ofInternational Fusion Energy Research Centre (IFERC-CSC),Aomori, Japan, under the Broader Approach collaboration be-tween Euratom and Japan, implemented by Fusion for Energyand JAEA.

a Poisson equation (or its asymptotic counterpart, the 17

so-called quasi-neutrality equation). The work presents 18

here will still be applicable taking into account electro- 19

magnetic effects. Since the Poisson equation is defined on 20

the particle coordinates, solving the gyrokinetic Vlasov- 21

Poisson system requires an operator that transforms the 22

gyro-center phase space in the particles phase space. This 23

operator is the so-called gyroaverage operator. We refer to 24

an abundant literature around this subject (see [3–5] and 25

references therein). 26

The present work is devoted to the numerical compu- 27

tation of the gyroaverage operator for a polar mesh which 28

is the framework in the gyrokinetic code Gysela [6,7]. We 29

follow a previous work which was designed for cartesian 30

geometry [1]. 31

We propose an alternative to the classical Pade ap- 32

proximation, that is especially employed in the gyrokinetic 33

code Gysela [6,7] and which is known to be only valid for 34

small Larmor radius. 35

The method is based on direct integration and interpo- 36

lation. Close to the method used already in the Gene code 37

(see [8,9]), it is applied in the context of 2D interpolation, 38

as we consider here a fully global simulation. 39

The paper is organized as follows. In Section 2, we in- 40

troduce the gyroaverage operator. The classical Pade ap- 41

proximation for the gyroaverage computation is described 42

in Section 3. In Section 4, we present the method based 43

Page 2 of 16

on interpolation. A numerical comparison with analyt-1

ical solutions is performed in Section 5. Finally, these2

gyroaverage operators will be applied to gyrokinetic sim-3

ulations in Section 6.4

2 Gyroaverage operator definition5

Let ρ be the gyro-radius which is transverse to b = B/B(with B the magnetic field) and which depends on thegyrophase α ∈ [0, 2π], i.e.

ρ = ρ(cos(α)e⊥1 + sin(α)e⊥2)

Here e⊥1 and e⊥2 are the unit vectors of a cartesian basis6

in the plane perpendicular to the magnetic field direction7

b. Let xG be the guiding-center radial coordinate and x8

the position of the particle in the real space. These two9

quantities differ by a Larmor radius ρ, i.e. x = xG + ρ.10

Let f : (r, θ) ∈ R+ × R �→ f(r, θ) be a polar function

and g : (x1, x2) ∈ R2 �→ g(x1, x2) the function defined by

g(r cos(θ), r sin(θ)) = f(r, θ) for all (r, θ). The function f(resp. g) stands for field quantity defined on x with polar(resp. cartesian) coordinates. The gyroaverage Jρ(f) of fdepending on the spatial coordinates is defined as:

Jρ(f)(r, θ) =12π

∫ 2π

0

g(xG + ρ)dα,

where xG = r(cos(θ), sin(θ)). This gyroaverage process11

consists in computing an average on the Larmor circle.12

It tends to damp any fluctuation which develops at sub-13

Larmor scales.14

Introducing f(k) the Fourier transform of f , with k =15

k(cos(θ), sin(θ)) the wave vector, then the operation of16

gyroaverage reads17

Jρ(f)(xG) =∫ 2π

0

dα

2π

∫R2

d2k

(2π)3f(k) exp{ik · (xG + ρ)}18

=∫

R2

d2k

(2π)3

[∫ 2π

0

dα

2πexp(ik⊥ρ cosα)

]19

×f(k) exp(ik · xG)20

where k⊥ is the norm of the transverse component of thewave vector k⊥ = k−(b.k)b. Let n a non-negative integerand consider Jn the Bessel function of the first kind andof order n, i.e ∀z ∈ C,

Jn(z) =i−n

π

∫ π

0

exp(iz cos θ) cos(nθ) dθ.

Therefore, the previous gyroaverage operation can be ex-21

pressed as a function of the Bessel function of first order22

J0 as:23

Jρ(f)(xG) =∫ +∞

−∞

d3k

(2π)3J0(k⊥ρ)f(k)eik·xG . (1)

Let consider a uniform polar mesh (r, θ) ∈ [rmin, rmax] ×[0, 2π[ with Nr × Nθ cells, our goal is to approximate theoperator

(fj,k) ∈ R(Nr+1)×Nθ �→ (Jρ(f)j,k) ∈ R

(Nr+1)×Nθ .

Nr + 1 points are needed to define Nr cells in r-direction 24

and Nθ points are sufficient to define Nθ cells in θ- 25

direction thanks to periodicity conditions. Considering ex- 26

pression (1), in Fourier space the gyroaverage is reduced to 27

the multiplication by a Bessel function of argument k⊥ρ. 28

Indeed, the Fourier transform of Jρ(f) can be written as: 29

Jρ(f)(k) =12π

∫R2

∫ 2π

0

f(xG + ρ)dα e−ixG·kdxG 30

=12π

∫ 2π

0

∫R2

f(xG+ρ)e−i(xG+ρ)·kdxG eiρ·kdα 31

=(

12π

∫ 2π

0

eikρ cos(α−θ)dα

)f(k) 32

which leads to 33

Jρ(f)(k) = J0(kρ)f(k). (2)

This operation is straightforward in simple geometry with 34

periodic boundary conditions, such as in local codes. Con- 35

versely, in the case of global codes, the use of Fourier 36

transform is not applicable for two main reasons: (i) ra- 37

dial boundary conditions are non periodic; and (ii) the ra- 38

dial dependence of the Larmor radius has to be accounted 39

for. Several approaches have been developed to overcome 40

this difficulty. The most widespread method for this gyro- 41

averaging process is to use a quadrature formula. In this 42

context, the integral over the gyro-ring is usually approx- 43

imated by a sum over four points on the gyro-ring [10]. 44

This is rigorously equivalent to considering the Taylor ex- 45

pansion of the Bessel function at order two in the small ar- 46

gument limit, namely J0(k⊥ρ) � 1−(k⊥ρ)2/4, and equiva- 47

lent to computing the transverse Laplacian at second order 48

using finite differences. This method has been extended 49

to achieve accuracy for large Larmor radius [11], i.e. the 50

number of points (starting with four) is linearly increased 51

with the gyro-radius to guarantee the same number of 52

points per arclength on the gyro-ring. In this approach 53

– used e.g. in [12,13] – the points that are equidistantly 54

distributed over the ring are rotated for each particle (or 55

marker) by a random angle calculated every time step. 56

This is performed on a finite element formalism and en- 57

ables therefore high order accuracy by keeping the matrix 58

formulation. 59

In reference [1] the influence of the interpolation op- 60

erator (which is of great importance when the quadra- 61

ture points do not coincide with the grid points) has 62

been studied and has shown that the cubic splines are 63

a good candidate. Some techniques used in reference [1] 64

taking advantages of the cartesian coordinates properties 65

are no more valid in (r, θ) polar geometry. In this pa- 66

per we present a method based on a direct integration 67

of the gyroaverage operator which is directly applicable 68

for global gyrokinetic code in toroidal geometry as Gysela 69

code. This new approach is tested for two different in- 70

terpolation methods, one based on cubic splines and the 71

other one on Hermite polynomials. Both are compared to 72

the Pade approximation. 73

Page 3 of 16

3 Method based on Pade approximation1

One possible solution to compute the gyroaverage oper-ator is to approximate the Bessel function with a Padeexpansion JPade(kρ) = 1/

[1 + (kρ)2/4

](e.g. see [14]). As

described in the following, such a Pade representation thenrequires the inversion of the Laplacian operator in realspace. Indeed, using this approximation, the relation (2)reads (

1 +(kρ)2

4

)Jρ(f)(k) = f(k)

which corresponds in real space to(

1 − ρ2

4Δ

)Jρ(f)(x) = f(x).

We project f and J(f) in the Fourier basis:2

f(r, θ) ≈Nθ−1∑n=0

An(r)einθ , Jρ(f)(r, θ) ≈Nθ−1∑n=0

Bn(r)einθ .3

The Laplacian in polar coordinates is expressed as:

Δ =∂2

∂r2+

1r

∂

∂r+

1r2

∂2

∂θ2

then we get for n = 0 . . .Nθ − 14

− ρ2

4B′′

n(r) − ρ2

4rB′

n(r) +(

1 +ρ2n2

4r2

)Bn(r) = An(r)5

that can be solved by finite differences. For the results pre-6

sented in the following, we use finite differences of second7

order which leads to a tridiagonal system.8

Remark 1. In case of a non-constant ρ, we also tested9

the operator 1 − ρ4∇ · (ρ∇); we found that the operator10

1 − ρ2

4 Δ gives better results.11

This Pade approximation gives the correct limit in12

the large wavelengths limit kρ 1 (see Fig. 1), while13

keeping JPade finite in the opposite limit kρ → ∞. The14

drawback is an overdamping of small scales: in the limit15

of large arguments x → ∞, JPade(x) → 4/x2, whereas16

J0 → (2/πx)1/2 cos(x − π/4) (see Fig. 2). Figure 2b17

presents, in real space, the exact gyroaverage of a 1D18

random function (in red) and its Pade approximation (in19

blue). The blue curve is near to the exact solution but20

doesn’t fit it completely.21

The method proposed in the next section which is no22

more based on an approximation of the Bessel function23

but on the direct calculation of the integral on a circle of24

radius ρ has been developed to overcome this drawback.25

4 Method based on interpolation26

In this section, we describe the computation of the gy-27

roaverage operator in real space. This method implies es-28

sentially interpolations over the Larmor circle. We put N29

uniformly distributed points on the circle of integration30

Fig. 1. The zero-th order Bessel function J0(kρ) compare toits Pade approximation 1/

[1 + (kρ)2/4

].

(a) (b)

Fig. 2. Exact and approximated gyroaverage operators ap-plied on an arbitrary function Fk exhibiting a broad spectrumranging from low to large wavelengths as compared with theLarmor radius ρ: (a) representation in the Fourier space, (b)representation in the real space (figures from [14]).

and we approximate the function value at these points 31

by interpolation. The gyroaverage is then obtained by the 32

rectangle quadrature formula on these points. More pre- 33

cisely, for a given point (rj , θk), the gyroaverage at this 34

point is approximated by: 35

36

Jρ(f)j,k � 37

12π

N−1∑�=0

P(f)(rj cos θk + ρ cosα�, rj sin θk + ρ sin α�)Δα, 38

39

where α� = �Δα and Δα = 2π/N . Since the quadrature 40

points do not coincide with grid points, we introduce an 41

interpolation operator P which can be 42

– Hermite interpolation; 43

– cubic splines interpolation. 44

Page 4 of 16

•

Fig. 3. Scheme of the method based on interpolations. Angyroaverage based on 5 points of interpolation is shown as ex-ample. The black point which is on the mesh and which is thecenter of the circle is the point where the gyroaverage is done.The blue points represent the positions where the given fieldmust be interpolated.

When points are outside the domain in the previous sum,1

we make a radial projection on the border of the domain.2

More precisely,3

– if r < rmin then P(f)(r, θ) will be replaced by4

P(f)(rmin, θ);5

– if r > rmax then P(f)(r, θ) will be replaced by6

P(f)(rmax, θ).7

Remark 2. In applications to gyrokinetic simulations, the8

gyroaverage operator is applied to f − feq where f is the9

distribution function and feq is the distribution function10

at equilibrium. The function f − feq has values close to 011

at the domain boundaries.12

As detailed in reference [15], the interpolation step canbe reformulated into a matrix-vector product,

P(f)(rj cos θk + ρ cosα�, rj sin θk + ρ sin α�) = (A�c)j,k,

where � stands for the index of the point where we makethe interpolation (blue points in Fig. 3), c denotes thesplines coefficients (cubic splines method) or the functionvalues (Hermite method). Note that the vector c has twoindex j, k as c is a vector containing data from the entirepolar grid. The size of this vector, depending on the case,is given below. So, the gyroaverage can be itself viewed asa matrix-vector product

Jρ(f)j,k =12π

N−1∑�=0

(A�f)j,kΔα = (Aρc)j,k.

As a consequence, for a given Larmor radius ρ, the matrix13

Aρ can be stored once for all.14

For each method, 2 versions are implemented:15

– a basic version;16

– a version with precomputation where we first computethe matrix Aρ such that

(Jρ(f)j,k) = Aρc,

Table 1. Time (in s.) of Hermite precompute and Pade as afunction of ρ with different orders (4, 6, 10 or 18) for the recon-struction of the derivatives in Hermite interpolation case (seeRemark 4). Parameters: rmin = 0.1, rmax = 0.9, Nr = Nθ =512, N = 1024, 100 iterations of the gyroaverage. 100ρ/rmax

express the percentage of ρ w.r.t. the simulation radius rmax.

100ρ/rmax Hermite (4) (6) (10) (18) splines Pade0 6 7 9 14 8 0.6

0.1 10 11 13 18 12 21.1 40 41 43 48 53 211.1 446 442 448 453 594 2

Table 2. Time (in s.) as a function of ρ for Hermite with-out and with precomputing. Parameters: rmin = 0.1, rmax =0.9, Nr = Nθ = 128, N = 1024, order of interpolation: 4, 100 it-erations of the gyroaverage.

100ρ/rmax Hermite basic Hermite precompute0 301 0.6

0.1 316 0.81.1 312 111.1 304 5

where c are spline coefficients (vector of size (Nr + 17

1)Nθ) or function values and derivatives in the case of 18

Hermite interpolation (size is 4(Nr + 1)Nθ). 19

Remark 3. Note that for the Hermite interpolation, we 20

need first to compute the derivatives at each cell interface. 21

These derivatives are reconstructed by centered finite dif- 22

ferences of arbitrary even order (we could also use odd 23

order by having only a C0 reconstruction, but then the 24

size of c would increase to 9(Nr +1)Nθ). In the numerical 25

results, we will take the order 4. Execution times to com- 26

pute the gyroaverage with different interpolation meth- 27

ods are given in Table 1. We observe that in the case of 28

Hermite interpolation, the order doesn’t impact too much 29

the performances. 30

Remark 4. Other time comparison are given in Tables 1 31

and 2. The use of the precomputation version is here quite 32

efficient, as the Larmor radius ρ is fixed and the matrix 33

is the same for each value of θ, which implies that the 34

storage is reduced; but the basic version permits to give a 35

rough indication of the time that would be used for more 36

general situations (where for example the storage would 37

be an issue), that are not considered for the moment. 38

Remark 5. If we compare the numerical cost of the 39

Hermite method and the Pade approximation (see Tab. 1), 40

we observe that the Hermite one needs a lot of more time, 41

especially for large radius. This fact is not an issue because 42

in many applications, which use the gyroaverage opera- 43

tor, are at least of dimension 4, whereas the gyroaverage 44

is applied only in 3D. As an example, we have obtained 45

the following times in the case of a drift kinetic simulation 46

(see Sect. 6.1): 42 s. for PADE, 43 s. for Hermite interpola- 47

tion with precomputation (using 1024 quadrature points). 48

Without precomputation, the time for Hermite is 47 s. 49

with 16 quadrature points and 270 s. with 1024 quadra- 50

ture points. Computations are made on a local cluster 51

Page 5 of 16

of the University of Strasbourg using 16 processors (grid1

32 × 32 × 32 × 64, 100 iterations). Further time measures2

will be detailed in Section 6.1.3

Remark 6. This method, which reduces to a simple in-4

terpolation, is very flexible because it does not depend5

on the mesh. It can also be easily extended to arbitrary6

coordinates.7

5 Numerical comparison with analytical8

solutions9

5.1 Definition of a class of analytical solutions10

depending on boundary conditions11

First, we give a family of functions whose gyroaverage is12

analytically known. For these functions, we obtain the gy-13

roaverage just by multiplying them by the Bessel function.14

Let m ≥ 0 be an integer and Cm be the Bessel function15

of the first kind (denoted by Jm) or the Bessel function of16

the second kind (denoted by Ym), see [16]. The following17

proposition gives the analytic expression of the gyroaver-18

age of Fourier-Bessel type functions.19

Proposition. Let z ∈ C. The gyroaverage of

f(r, θ) = Cm(zr)eimθ

readsJρ(f)(r0, θ0) = J0(zρ)Cm(zr0)eimθ0 .

Proof: By definition,20

21

Jρ(f)(r, θ) =12π

∫ 2π

0

∫ +∞

0

∫ 2π

0

Cm(zr0)eimθ022

× δ{x0=x+ρ}dαdr0dθ02324

where x0 = r0(cos(θ0), sin(θ0)), x = r(cos(θ), sin(θ)) and25

ρ = ρ(cos(α), sin(α)).26

The additivity theorem of Graf for Bessel functions27

(see [17]) states that if u, v and w are the lengths of a tri-28

angle and α, γ the angles as shown in the following figure:29

u

w

v

α

γ

30

then for all integers m and all complex numbers z,

Cm(zw)eimγ =∞∑

k=−∞Cm+k(zu)Jk(zv)eikα.

Then, we obtain, with v = ρ, w = r0, u = r, γ = θ0 − θ 31

and α = α, 32

Jρ(f)(r, θ) 33

= eimθ 12π

∫ 2π

0

∫ +∞

0

∫ 2π

0

Cm(zr0)eim(θ0−θ)34

×δ{x0=x+ρ}dαdr0dθ0 35

= eimθ 12π

∫ 2π

0

∫ +∞

0

∫ 2π

0

( ∞∑k=−∞

Cm+k(zr)Jk(zρ)eikα

)36

×δ{x0=x+ρ}dαdr0dθ0 37

= eimθ

( ∞∑k=−∞

Cm+k(zr)Jk(zρ)

)38

× 12π

∫ 2π

0

∫ +∞

0

∫ 2π

0

eikαδ{x0=x+ρ}dαdr0dθ0 39

= eimθ

( ∞∑k=−∞

Cm+k(zr)Jk(zρ)

)× 1

2π

∫ 2π

0

eikαdα. 40

We use the fact that

12π

∫ 2π

0

eikαdα = δk,0

in order to conclude that

Jρ(f)(r, θ) = J0(zρ)Cm(zr)eimθ. �

In the following, we give some examples of these test func- 41

tions depending on the boundary conditions we want to 42

test. 43

Examples 44

1. rmin = 0 and homogeneous Dirichlet condition on 45

rmax. 46

Here we consider a disk [0, rmax]× [0, 2π] and the func-tion

f1(r, θ) = Jm

(r

jm,�

rmax

)eimθ

where jm,� is the �th zero of Jm. The function f1 ver-ifies the following Dirichlet condition:

f1(rmax, θ) = 0, 0 ≤ θ < 2π,

and its gyroaverage reads

Jρ(f1)(r0, θ0) = J0

(ρ

jm,�

rmax

)f1(r0, θ0).

2. Homogeneous Dirichlet conditions on rmin > 0 and 47

rmax. 48

The following function is defined on the annulus 49

[rmin, rmax] × [0, 2π] 50

f2(r, θ) =(

Jm(γm,�)Ym

(rγm,�

rmax

)51

− Ym(γm,�)Jm

(rγm,�

rmax

))eimθ

52

Page 6 of 16

where γm,� is the �th zero of

y �→ Jm(y)Ym

(y

rmin

rmax

)− Ym(y)Jm

(y

rmin

rmax

).

The function f2 verifies the following Dirichlet condi-tions:

f2(rmin, θ) = 0, f2(rmax, θ) = 0, 0 ≤ θ < 2π,

and its gyroaverage reads

Jρ(f2)(r0, θ0) = J0

(ρ

γm,�

rmax

)f2(r0, θ0).

3. Homogeneous Neumann conditions on rmin > 0 and1

rmax.2

The following function is defined on the annulus3

[rmin, rmax] × [0, 2π]4

f3(r, θ) =(

J ′m(ηm,�)Ym

(r

ηm,�

rmax

)5

− Y ′m(ηm,�)Jm

(r

ηm,�

rmax

))eimθ

6

where ηm,� is the �th zero of

y �→ J ′m(y)Y ′

m

(y

rmin

rmax

)− Y ′

m(y)J ′m

(y

rmin

rmax

).

The function f3 verifies the following Neumann condi-tions:

∂rf3(rmin, θ) = 0, ∂rf3(rmax, θ) = 0, 0 ≤ θ < 2π,

and its gyroaverage reads

Jρ(f3)(r0, θ0) = J0

(ρ

ηm,�

rmax

)f3(r0, θ0).

We have here used the fact that for Cn = Jn or Yn, thederivative reads:

C′n(r) = −Cn+1(r) +

nCn(r)r

.

We show in Figure 4 the real and imaginary parts of thefunction

(r, θ) ∈ [0, 5]× [0, 2π] �→ J1(r)eiθ .

5.2 Numerical results7

In this part, the different numerical methods are compared8

in the case of the second test case (Homogenous Dirichlet9

condition on rmin and rmax) with rmin = 0.1, rmax = 0.9,10

� = 1 and m = 1, 5, 20, 60. We consider Nr = Nθ = 51211

in Tables 3–8 and Nr = Nθ = 1024 in Tables 9–10. In12

Tables 3 and 4, we give here the L2-norm error of the13

gyroaverage function for m = 1 whereas Tables 5, 6 refer14

to m = 5, Tables 7, 8 refer to m = 20 and Tables 9, 1015

refer to m = 60. In Table 11, we use various orders of16

interpolation for Hermite. Note that for each method, the17

error is computed over the domain [rmin + ρ, rmax − ρ].18

Table 3. Comparison Pade with Hermite (m = 1).

100ρ/rmax Pade N = 4 8 16 1024

0 10−17 10−17 10−17 10−17 10−17

0.1 4 × 10−11 3 × 10−6 3 × 10−6 3 × 10−6 3 × 10−6

1.1 2 × 10−8 3 × 10−6 1 × 10−6 4 × 10−7 5 × 10−7

11.1 2 × 10−4 3 × 10−4 1 × 10−5 4 × 10−8 5 × 10−8

Table 4. Comparison Pade with splines (m = 1).

100ρ/rmax Pade N = 4 8 16 1024

0 10−17 10−17 10−17 10−17 10−15

0.1 4 × 10−11 3 × 10−6 3 × 10−6 3 × 10−6 3 × 10−6

1.1 2 × 10−8 3 × 10−6 1 × 10−6 5 × 10−7 6 × 10−7

11.1 2 × 10−4 3 × 10−4 1 × 10−5 4 × 10−8 9 × 10−8

Table 5. Comparison Pade with Hermite (m = 5).

100ρ/rmax Pade N = 4 8 16 1024

0 10−17 10−17 10−17 10−17 10−17

0.1 1 × 10−8 5 × 10−6 5 × 10−6 5 × 10−6 5 × 10−6

1.1 2 × 10−6 2 × 10−6 1 × 10−6 8 × 10−7 7 × 10−7

11.1 1 × 10−3 1 × 10−3 3 × 10−5 1 × 10−7 6 × 10−8

Table 6. Comparison Pade with splines (m = 5).

100ρ/rmax Pade N = 4 8 16 1024

0 10−17 10−17 10−17 10−17 10−17

0.1 1 × 10−8 5 × 10−6 5 × 10−6 5 × 10−6 5 × 10−6

1.1 2 × 10−6 2 × 10−6 1 × 10−6 8 × 10−7 8 × 10−7

11.1 1 × 10−3 1 × 10−3 3 × 10−5 2 × 10−7 1 × 10−7

Table 7. Comparison Pade with Hermite (m = 20).

100ρ/rmax Pade N = 4 8 16 1024

0 10−18 10−17 10−17 10−17 10−17

0.1 1 × 10−8 5 × 10−6 4 × 10−6 4 × 10−6 4 × 10−6

1.1 6 × 10−6 1 × 10−6 1 × 10−6 7 × 10−7 7 × 10−7

11.1 9 × 10−3 3 × 10−3 1 × 10−5 9 × 10−8 6 × 10−8

Table 8. Comparison Pade with splines (m = 20).

100ρ/rmax Pade N = 4 8 16 1024

0 10−18 10−17 10−17 10−17 10−16

0.1 1 × 10−8 4 × 10−6 4 × 10−6 4 × 10−6 4 × 10−6

1.1 6 × 10−6 1 × 10−6 1 × 10−6 8 × 10−7 8 × 10−7

11.1 9 × 10−3 3 × 10−3 1 × 10−5 1 × 10−7 1 × 10−7

Table 9. Comparison Pade with Hermite (m = 60).

100ρ/rmax Pade N = 4 8 16 1024

0 10−18 10−17 10−17 10−17 10−17

0.1 2 × 10−8 7 × 10−7 2 × 10−7 2 × 10−7 2 × 10−7

1.1 9 × 10−5 1 × 10−5 6 × 10−7 2 × 10−7 4 × 10−7

11.1 1 × 10−3 2 × 10−3 2 × 10−3 3 × 10−7 1 × 10−8

Page 7 of 16

Fig. 4. Real and imaginary parts of the test function (r, θ) �→J1(r) exp(iθ).

Table 10. Comparison Pade with splines (m = 60).

100ρ/rmax Pade N = 4 8 16 1024

0 10−18 10−17 10−17 10−17 10−16

0.1 2 × 10−8 9 × 10−7 2 × 10−7 2 × 10−7 2 × 10−7

1.1 9 × 10−5 1 × 10−5 6 × 10−7 1 × 10−7 5 × 10−7

11.1 1 × 10−3 2 × 10−3 2 × 10−3 3 × 10−7 2 × 10−8

Table 11. Comparison Hermite with different interpolationorder. Parameters: N = 1024, m = 1.

100ρ/rmax Hermite(4) (6) (10) (18)

0 10−17 10−17 10−17 10−17

0.1 3 × 10−6 3 × 10−6 3 × 10−6 3 × 10−6

1.1 5 × 10−7 6 × 10−7 6 × 10−7 6 × 10−7

11.1 5 × 10−8 8 × 10−8 1 × 10−7 1 × 10−7

Remark 7.1

1. For ρ = 0, all the methods are exact.2

2. The method based on interpolation gives almost the3

same results with Hermite interpolation or cubic4

splines interpolation.5

3. The method based on Pade approximation gives very6

good results for small values of ρ.7

4. For large values of ρ, the method based on inter-8

polation gives better results even with a relatively9

small number of points on the circle (for example with10

ρ = 0.1 and N ≥ 8).11

5. The value N = 1024 is selected to have a converged12

value of the gyroaverage. We see that very often N =13

16 points are already sufficient to obtain a very good14

approximation of this converged value.15

6. In Tables 3 and 4, the error is sometimes larger for16

N = 1024 than for N = 16. This could be explained by17

the fact that the 16 points are closer to the mesh points18

and the interpolation error in (r, θ) is greater than the19

interpolation error on the circle of integration.20

7. In the case of high mode, we need more points on the21

circle in order to evaluate the gyroaverage when the22

radius is large (Tabs. 9 and 10).23

8. In the case of Hermite interpolation, we need to24

compute the derivatives at each cell interface. These25

derivatives are reconstructed by centered finite differ-26

ences of arbitrary even order. In Table 11, these or-27

ders are 4, 6, 10 or 18. When this order is not specified28

(Tabs. 3, 5, 7 and 9), the default value is 4. Table 11 29

shows that it does not get better results by increasing 30

this order of reconstruction. 31

6 Application to gyrokinetic simulations 32

The computational effort to numerically solve the 6 33

dimensional Vlasov-Maxwell systems which describes 34

plasma turbulence in tokamak plasmas still remains out 35

of reach for present day supercomputers. All the numer- 36

ical simulations performed until now in this domain take 37

care of the gyrokinetic ordering to reduce this problem 38

of one dimension. This ordering take into account the 39

fact that (i) electromagnetic fluctuations occur on time 40

scales much longer than charged particle gyration period 41

(ω/Ωc 1 with ω the fluctuation frequency and Ωc 42

the cyclotron frequency); and (ii) the wavelength of these 43

fluctuations is much smaller than the characteristic scale 44

length of the gradients of magnetic field, density and tem- 45

perature. See [18] for a detailed review on the gyrokinetic 46

framework and simulations to compute turbulent trans- 47

port in fusion plasmas. Within this gyro-ordering, the so- 48

called gyrokinetic model can be derived (see [10]) by av- 49

eraging on the fast gyration of charged particles around 50

the magnetic field lines. The magnetic toroidal configura- 51

tion considered in this paper is simplified. Indeed, mag- 52

netic flux surfaces are assumed concentric tori with cir- 53

cular cross-sections. The gyroaverage operator described 54

in Section 2 occurs in this reduction from 6 to 5 dimen- 55

sions. The new 5D set of coordinates corresponds to: (i) 56

3D toroidal spatial coordinates (r, θ, ϕ) (with r the radial 57

direction; θ and ϕ the poloidal (resp. toroidal) angle); and 58

(ii) 2D in velocity space with v‖ the velocity parallel to 59

the magnetic field line and μ = mv2⊥/(2B) the magnetic 60

moment where v⊥ represents the velocity in the plane or- 61

thogonal to the magnetic field. It is important to note that 62

in this ordering μ is an adiabatic invariant, so it plays the 63

role of a parameter in the 5D gyrokinetic Vlasov equation. 64

In the following, the 4D problem which is treated in 65

Section 6.1 corresponds to the case where we consider a 66

unique value of μ, i.e. the same Larmor radius is taken for 67

all particles. The 5D problem, we deal with in Section 6.2, 68

several values of μ considered to take into account the 69

dependence on the Larmor radius with v⊥. 70

The time evolution of the gyrocenter distribution func- 71

tion f is given by the gyrokinetic conservative equation 72

(see also Eqs. (17)–(20) in [18]): 73

B∗‖∂f

∂t+∇∇∇ ·

(B∗

‖dxG

dtf

)+

∂

∂vG‖

(B∗

‖dvG‖dt

f

)= 0 (3)

where xG and vG‖ are respectively the space coordinates 74

and the parallel velocity of the guiding centers. In the 75

electrostatic limit, for a particle of mass m and charge q 76

the motion equations of the guiding centers are given by 77

dxG

dt= vG‖b∗ + vE×B + vD (4) 78

mdvG‖dt

= −μ∇∇∇∗‖B − q∇∇∇∗

‖Φ + mvG‖vE×B · ∇∇∇B

B(5) 79

Page 8 of 16

where ∇∇∇∗‖ ≡ b∗ · ∇∇∇, while b∗ and B∗

‖ are defined by:1

b∗ ≡ BB∗

‖+

mvG‖qB∗

‖B∇∇∇× B (6)2

B∗‖ ≡ B +

mvG‖qB

b · (∇∇∇× B) . (7)3

The ‘E×B’ drift is equal to vE×B = (1/B∗‖)b×∇∇∇Φ while4

curvature drift is defined as vD = (mv2

G‖+μB

qB∗‖

)b × ∇∇∇BB .5

We focus on the turbulent transport driven by col-6

lisionless ITG instability so electrons are assumed adia-7

batic. In this limit, the gyroaveraged electrostatic poten-8

tial Φ (equivalent to J√2μΦ notation) is solution of the9

self-consistently coupled 3D quasi-neutrality equation:10

−(

∂2rΦ +

(1r

+∂rn0(r)n0(r)

)∂rΦ +

1r2

∂2θΦ

)

+1

Te(r)(Φ − λ〈Φ〉) =

1n0(r)

J√2μ

(∫f − feqdv

)

(8)

〈Φ〉 =1L

∫ L

0

Φ(r, θ, z)dz (9)

where Te and n0 are electron temperature and density11

profiles which will be defined latter.12

λ is an integer which is expand to 0 or 1 according to13

the fact that zonal flows are token into account or not.14

The new gyroaverage methods presented before have15

been tested with two codes: (i) the SELALIB plate-16

form [19] for the 4D simplified case; and (ii) the Gysela17

code [7] for the benchmark with the classical Cyclone18

DIII-D 5D case. Both are based on a classical Backward19

semi-Lagrangian scheme (BSL) with cubic splines inter-20

polation, predictor-corrector method.21

In the following numerical solutions are computed us-22

ing normalized equations. The temperature is normalized23

to Te0, where Te0 is defined by the initial temperature24

profile such that Te(rp)/Te0 = 1 where rp ∈ [rmin, rmax]25

is the radial coordinate of the peak of the initial distri-26

bution function. The time is normalized to the inverse of27

the ion cyclotron frequency ωc = eiB0/mi. Velocities, in-28

cluding the parallel velocity, are expressed in units of the29

ion speed vT0 =√

Te0/mi, the electric potential is nor-30

malized to Te0/ei and the magnetic field is normalized to31

B0. Consequently, lengths are normalized to the Larmor32

radius ρ = mivT0/eiB0 and the magnetic moment μ to33

Te0/B0.34

6.1 Simplified 4D SLAB case35

In this section, we consider a simplified model of the sys-36

tem of equations (3)–(9). A periodic cylindrical plasma of37

radius a and length 2πR (with R the major radius) is con-38

sidered as a limit case of a stretched torus. The plasma is39

confined by a strong magnetic which is uniform B = Bez40

where ez stand for the unit vector in the toroidal direction 41

z. With theses assumptions the velocity drifts are reduced 42

to the E × B drift. This SLAB 4D case is equivalent to 43

the one treated in [20] or [6]. 44

The equation satisfied by the distribution function of 45

ions f(t, r, θ, z, v) following the guiding center movement 46

reads: 47

∂tf −(

∂θJ√2μΦ

r

)∂rf +

(∂rJ√

2μΦ

r

)∂θf

+ v∂zf − (∂zJ√

2μΦ)∂vf = 0 (10)

for (r, θ, z, v) ∈ [rmin, rmax]×[0, 2π]×[0, L]×[−vmax, vmax]. 48

To deals with this equation system, we have used the 49

SELALIB platform [19] with a classical semi-Lagrangian 50

method with cubic splines interpolation, predictor correc- 51

tor method and Verlet algorithm for the characteristics 52

(see [21] for details). The platform has been improved by 53

adding the one fixed μ capability and by implementing 54

the three different gyroaverage operators described in Sec- 55

tions 3 and 4. In our case the MPI parallelization is based 56

on transpositions between (r, θ, v) domain decomposition 57

and z domain decomposition. In this section the numerical 58

instability growth rates are compared to the one deduced 59

from the dispersion relation obtained by linearizing the 60

self-consistent equation system (9)–(10). 61

The derivation of the dispersion relation is given in Ap- 62

pendix A. We adapt a code available in SELALIB, that 63

computes the zeros for the dispersion relation (A.3), as in 64

reference [22], by adding the gyroaverage term. Figure 9 65

presents the instability rates as a function of μ. We ob- 66

tain the two first curves by solving the dispersion relation 67

with J20 (√

2μ) (curve in red) or by substituting J0(√

2μ) 68

by its Pade approximation (curve in green). We have cho- 69

sen κ = 1, in (A.3). The two remaining curves are obtain 70

numerically with the Pade method for the gyroaverage 71

operator (curve in blue) or the method with Hermite in- 72

terpolation (curve in magenta). It appears that the slope 73

decreases faster with the Hermite interpolation method 74

than with the Pade method. The slopes obtained with the 75

numerical Pade are different from these obtained with the 76

dispersion relation and the Pade approximation because 77

the functions that we consider here are not Fourier-Bessel 78

functions. 79

In the simulations, we will take λ = 0 (no zonal flow 80

case). The initial distribution function reads: 81

82

f(0, r, θ, z, v) = feq(r, v) 83

×(

1 + ε exp(− (r − rp)2

δr

)cos

(2πn

Lz + mθ

))84

85

where the equilibrium function feq is

feq(r, v) =n0(r) exp

(− v2

2Ti(r)

)

(2πTi(r))1/2.

The profiles Ti, Te and n0 are given by:

P(r) = CP exp(−κPδrP tanh

(r − rp

δrP

))

Page 9 of 16

Fig. 5. Linear growth rate versus kθρi for the CYCLONEDIII-D base case with: (circles) Pade gyroaverage operator and(triangles) gyroaverage using Hermite interpolation.

where P ∈ {Ti, Te, n0}, CTi = CTe = 1 and

Cn0 =rmax − rmin∫ rmin

rmaxexp

(−κn0δrn0 tanh

(r−rp

δrn0

))dr

.

We consider the parameters of [22] [Medium case]:1

rmin = 0.1, rmax = 14.5, vmax = 7.32, κn0 = 0.055,2

κTi = κTe = 0.27586, δrTi = δrTi =δrn0

2= 1.45,3

ε = 10−6, n = 1, m = 5,4

L = 1506.759067, rp =rmin + rmax

2, δr =

4δrn0

δrTi

.5

Numerical results are given in Figures 6–10. We will con-6

sider here N = 1024 for Hermite, with precomputation.7

In Figure 4, we plot a poloidal cut f(r, θ, 0, 0) (μ = 0.5, 1)8

at time 7000 for Pade and Hermite. We observe smaller9

structures for Pade, indicating that the instability occurs10

in a more faster way. Similar comments can be done for11

Figure 5 in which the same poloidal cut is plotted, but12

for μ = 0, 0.1. Obviously, when μ = 0, the two methods13

give the same results and for μ = 0.1, the two methods14

give very comparable results, which is in a good agree-15

ment with the dispersion relation computations. Finally,16

Figure 6 gathers (for a less refined mesh 644 and Δt = 5)17

the poloidal cut for different values of μ (μ = 0.1, . . . , 0.8).18

Differences between Pade and Hermite become more and19

more significant when μ increases (see in particular, the20

case μ = 0.8). Figure 9 shows the time evolution of21 ∫ rmax

rmin

∫ 2π

0 Φ(r, θ, 0)rdrdθ in the linear phase. The gyroav-22

erage tends to reduce the instability rate; the more μ is23

large, the more this rate is small. The comparison with the24

solution of the dispersion relation confirms that the insta-25

bility rate is lower in the Hermite interpolation case. We26

present the time evolution of∫ rmax

rmin

∫ 2π

0Φ(r, θ, 0)rdrdθ in27

Fig. 6. Poloidal cut f(r, θ, z = 0, v‖ = 0) at time T = 7000 for128×256×128×128, Δt = 2, successively from top to bottomμ = 0.5 with Hermite and then Pade; μ = 1, with Hermite andthen Pade.

Page 10 of 16

Fig. 7. Poloidal cut f(r, θ, z = 0, v‖ = 0) at time T = 5000 for128×128×128×128, Δt = 1, successively from top to bottomμ = 0.1 with Hermite and then Pade; μ = 0, with Hermite andthen Pade.

Fig. 8. Poloidal cut f(r, θ, 0, 0) at time T = 5000 for 64×64×32× 64, Δt = 5 (left, Hermite; right, Pade) for μ = 0.1, . . . , 0.8(from top to bottom).

Page 11 of 16

Fig. 9. Time evolution of∫ rmax

rmin

∫ 2π

0Φ(r, θ, 0)rdrdθ: Hermite

(top), Pade (middle). Bottom: instability rates as a function ofμ; comparison between solution of dispersion relation (A.3)(using J0(

√2μ) or its Pade approximation) and numerical

results.

the nonlinear phase (Fig. 10). In this phase, the instabil-1

ity is growing faster in the Pade case than in the Hermite2

interpolation case. Moreover, when μ is small, one can see3

that the instability occurs in a more slower way.4

Time results (Tabs. 12 and 13) show that the choice5

of the gyroaverage operator is not very influential in the6

Fig. 10. Time evolution of∫ rmax

rmin

∫ 2π

0Φ(r, θ, 0)rdrdθ. Compar-

ison Hermite/Pade.

total time. Indeed, the computation of the gyroaverage 7

appears to be a 3D problem in a 4D environment. 8

6.2 Benchmark with the classical 5D Cyclone DIII-D 9

case 10

6.2.1 Numerical results obtained with Gysela code 11

For this part, the gyroaverage operator based on cubic 12

spline and Hermite interpolation have been implemented 13

in the Gysela code [7] and compared to the existing Pade 14

Page 12 of 16

Table 12. Time (in min.) on HPC, mesocenter of the Univer-sity of Strasbourg, with 64 processors (8 nodes). Nr × Nθ ×Nz × Nv = 64 × 64 × 32 × 64, Δt = 5, 1600 iterations.For Hermite, with precomputation: N = 1024. The values100ρ/rmax = 100

√2μ/rmax, which express the percentage of

ρ w.r.t. the simulation radius, are given for correspondancewith the previous section on analytical results.

μ 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9100ρ/rmax 3.1 4.4 5.3 6.2 6.9 7.6 8.2 8.7 9.3

Herm. 15 16 15 16 16 16 16 16 15Pade 15 15 14 15 17 15 16 16 15

Table 13. Final time (in s.) reached for a 24 hours simulationwith Nr ×Nθ ×Nz ×Nv = 128× 256× 128× 128, Δt = 2. OnHelios Computational Simulation Centre, International FusionEnergy Research Centre of the ITER Broader Approach. Su-percomputer with 8820 processors (4410 nodes; each node hav-ing 16 cores). For Hermite, with precomputation: N = 1024.

μ Hermite Pade0.5 12524 133261 12556 13454

approximation. The 5D Vlasov-Poisson problem consid-1

ered is the one described by equations (3)–(9). In the fol-2

lowing all the numerical results presented are expressed3

in normalized units of the code except in Figure 5. In4

Gysela code, the temperature is normalized to Te0, where5

Te0 is defined by the initial temperature profile such that6

Te(rp)/Te0 = 1. The time is normalized to the inverse of7

the ion cyclotron frequency ωc = eiB0/mi. Velocities, in-8

cluding the parallel velocity, are expressed in units of the9

ion speed vT0 =√

Te0/mi, the electric potential is nor-10

malized to Te0/ei and the magnetic field is normalized to11

B0. Consequently, lengths are normalized to the Larmor12

radius ρs = mivT0/eiB0 and the magnetic moment μ to13

Te0/B0.14

First of all, we compare the two methods from the15

numerical point of view. The numerical comparisons have16

been performed on a linear benchmark based on the classi-17

cal cyclone DIII-D case [2]. This kind of typical benchmark18

had already been performed several years ago to validate19

the Gysela code (cf. [23]). For the present tests, the same20

parameters than in Section 4 of [23] have been used ex-21

cept for the size of the simulation. In this benchmark, we22

consider the following mesh:23

Nr = 256, Nθ = 256, Nϕ = 64, Nv‖ = 64, Nμ = 8. (11)

For both Hermite and cubic spline methods, N = 3224

quadrature points are used. Five simulations have been25

performed for the three different gyroaverage operators26

(Pade, Hermite and cubic splines). Each corresponding27

to the exciting of a different initial unstable mode (m, n)28

with m the poloidal mode number and n the toroidal one.29

The results obtained with cubic spline interpolation are30

not detailed because they are very similar to the results31

obtained with the Hermite interpolation. The numerical32

Table 14. Linear mode growth rates normalized to ΩC0 forthe Cyclone DIII-D base case.

(m, n) (4,−3) (10,−7) (14,−10)

kθ

√(2μmax) 0.98 1.98 2.77

γ with Pade 1.3e−4 5.7e−4 7.9e−4

γ with Herm. 1.3e−4 5.6e−4 7.6e−4

(m, n) (17,−12) (21,−15)

kθ

√(2μmax) 3.36 4.15

γ with Pade 7.8e−4 6.2e−4

γ with Herm. 7.16e−4 5.2e−4

values of the normalized linear growth rates associated to 33

each (m, n) modes are given in Table 14. In Figure 5 the 34

results are plotted as proposed in Dimits paper (cf. Fig. 1 35

in [2]). 36

In these simulations, the minimum and maximum val- 37

ues of μ are respectively μmin = 0.143 and μmax = 7. Then, 38

using the relation kθ

√2μ ∼ m/rp

√2μ = 0.5a

√2μ with a 39

the radial size of the simulated box being equal to 100, 40

the range of the larmor radius values for the gyroaverage 41

operator are between 0.028 m√

μmin and 0.028 m√

μmax. 42

The maximum value for each mode (m, n) are given in 43

Table 14. We observe that results are similar between 44

Pade and Hermite for the modes (m, n) = (4,−3) and 45

(m, n) = (10,−7) and that the linear growth rates be- 46

come smaller with the Hermite gyroaverage operator for 47

m greater than 14. This is in agreement with what has 48

been observed in previous section and what is shown in 49

Figure 1. The Pade expansion approximates correctly the 50

Bessel function J0(kθρ) for kθρ < 1 but overestimate it 51

for 1 < kθρ < 5. Non-linear long time simulations should 52

be run in the future to analyze the impact on the large 53

Fourier wave numbers (kθρ > 5). 54

6.2.2 Parallel optimization of Hermite based gyroaverage 55

operator 56

Another issue for the Hermite or cubic spline operators 57

was to be competitive in terms of computational time with 58

the existing optimized Pade operator used until now for 59

Gysela simulations. A performance analysis is presented 60

in the following. 61

To compare the performance of the different gyroav- 62

erage methods, different test cases of 4 iterations were 63

done on the Helios machine. Computation nodes used dur- 64

ing our tests were equipped of two Intel Xeon E5-2450 65

2.10 GHz processors, so 16 cores a node. On Table 15, the 66

different test cases have been performed over the three 67

following meshes (12), (13) and (14): 68

Nr = 128, Nθ = 128, Nϕ = 32, Nv‖ = 16, Nμ = 4 (12)

Nr = 256, Nθ = 256, Nϕ = 32, Nv‖ = 16, Nμ = 4 (13)

Nr = 512, Nθ = 512, Nϕ = 32, Nv‖ = 16, Nμ = 4 (14)

Page 13 of 16

Table 15. Execution time and portion of the different versions of the gyroaverage operator over three different meshes.

Gyroaverage methodPade Hermite Spline Disable

Without With Without WithMesh (12): Total execution time (s) 26.71 46.24 29.00 47.70 28.72 26.33

Percentage of gyroavg. execution 1.4% 75.6% 10.2% 81.2% 9.1% ∅Mesh (13): Total execution time (s) 105.98 186.83 120.55 190.25 127.12 104.78

Percentage of gyroavg. execution 1.1% 78.3% 15.0% 81.6% 21.3% ∅Mesh (14): Total execution time (s) 453.52 760.85 525.18 793.47 594.18 444.20

Percentage of gyroavg. execution 2.1% 71.3% 18.2% 78.6% 33.8% ∅

Table 16. Over the Mesh 14: execution time in second, relative efficiency in percentage and portion of the different versions ofthe gyroaverage operator with a set of number of threads.

Gyroaverage methodPade Hermite Disable

#thread: 1 Total execution time (s) 454.74 526.75 443.85Relative efficiency 100.0% 100.0% ∅

Percentage of gyroavg. execution 2.5% 18.7% ∅#thread: 2 Total execution time (s) 232.53 265.67 227.86

Relative efficiency 97.8% 99.1% ∅Percentage of gyroavg. execution 2.1% 16.6% ∅

#thread: 4 Total execution time (s) 121.07 141.83 117.83Relative efficiency 93.9% 92.8% ∅

Percentage of gyroavg. execution 2.7% 20.4% ∅#thread: 8 Total execution time (s) 66.37 77.66 64.98

Relative efficiency 85.6% 84.8% ∅Percentage of gyroavg. execution 2.1% 19.5% ∅

One can notice that between the three meshes, the only1

difference is the number of points in radial and poloidal di-2

rections. The mesh (13) has four times more points than3

the mesh (12), and the same between the meshes (14)4

and (13). We choose to increase the number of points in5

these directions because the size of the poloidal cut has6

the most visible impact on the execution time of the gyro-7

average operator. To avoid the effect of the network on8

these measures of performance, the parameters of the dif-9

ferent runs were adjusted to fit only one compute node of10

the Helios1 machine. To simplify the interpretation of the11

results, the runs were done with only one thread.12

On Table 15, you can see the evaluation of 3 ver-13

sions of the gyroaverage operator: Pade, Hermite (without14

and with precomputation) and Spline (without and with15

precomputation). Also, the Gysela code can be executed16

without any gyroaverage at all. The execution time in this17

configuration allow us to deduce accurately the percentage18

of time that requires the gyroaverage operator.19

As expected, for all methods, the bigger the poloidal20

cut is, the more the portion of the execution time of the21

gyroaverage is consequent on the total time of the simu-22

lation. Over all the meshes, you can notice great acceler-23

ations of precomputation versions compared to versions24

without it in the case of Hermite and spline method.25

Although these accelerations, the methods based on in-26

1 http://www.top500.org/system/177449

terpolation are 9 times slower in average than the Pade 27

approximation. 28

For the last performance aspect of our study, we focus 29

on the impact of the number of threads. In fact, Gysela 30

code is parallelised in Mpi and OpenMP. The intent of this 31

hybrid parallelization is to use efficiently current super- 32

computer. In this context, the Hermite version of the gy- 33

roaverage has been upgraded to be callable concurrently 34

from multiple threads (thread safe). The effort of deve- 35

lopment were focused only on the Hermite based gyro- 36

average because it has at least two attractive aspects com- 37

pared to the Spline version: (i) our implementation of the 38

Hermite gyroaverage allow us to choice the degree of the 39

interpolation and so the possibility to improve the quality 40

interpolations; and (ii) the Hermite version requires only 41

a few points (depending on the degree) around the target 42

position of the interpolation whereas the Spline approach 43

requires the value of all points of the field to apply its 44

interpolation. 45

To assess the behavior of the gyroaverage, we made a 46

set of runs of 4 iterations with different number of threads 47

over the mesh (14). We keep a low number of iterations to 48

have reasonable times of execution. The runs were made 49

with 1, 2, 4 and 8 threads and respectively with 16, 32, 64 50

and 128 cores. 51

Table 16 contains different measures of performance in 52

three configurations of gyroaverage: (i) Pade, (ii) Hermite 53

with precomputation and (iii) gyroaverage disable. As for 54

Page 14 of 16

the previous process, the execution time and the portion1

of time of each gyroaverage are given. Also, the relative2

efficiency is given and computed thanks to:3

tref

#thtarg × ttarg× 100 (15)

where tref is the execution time of the reference run, for4

us the run with one thread, ttarg and #thtarg respectively5

the execution time and the number of threads of the target6

run. The higher the relative efficiency is, the better the7

quality of the parallelization is.8

First of all, you can notice the total of execution in the9

three configurations scales with the number of threads.10

Secondly, the Hermite and Pade methods have a sim-11

ilar behavior regarding the relative efficiency as the num-12

ber of threads increases. In the final case with 8 threads,13

both methods keep a relative efficiency above 80% which14

is quite good.15

Finally, if you pay attention to the portion of the ex-16

ecution time of a method for each number of threads,17

you can notice that it remains roughly constant. This18

means the gyroaverage operator scales with the number19

of threads as the time of execution scales.20

As in the previous analysis, in average the gyroaverage21

based on Hermite interpolations is 9 times slower than the22

Pade approximation.23

The previous performance analysis do not take into24

account the execution time of Gysela’s diagnostics. The25

gyroaverage operator has a great impact on their execu-26

tion time, but the measure of execution time of this part27

of the code is difficult because of the writing of output28

files on disk which can introduce an random behavior. In29

fact, some diagnostic applies a gyroaverage over the whole30

distribution function of guiding centers to obtain informa-31

tion about the distribution of particles. There is a room32

of improvement for the interpolation based gyroaverage33

operator to deal more efficiently this kind of operations.34

In the future, we intend to optimize the Hermite method35

in order to shorten its computation time and to be com-36

petitive with the Pade method.37

7 Conclusion38

We have validated the gyroaverage computation on po-39

lar geometry. Comparisons are made with classical Pade40

approximation, considering on the one hand analytical41

cases, for which we know the exact solution, and on the42

other hand some basic gyrokinetic simulations: a 4D drift-43

kinetic model with one Larmor radius and the classical44

linear DIII-D benchmark case.45

We find that, in the linear case, the differences with46

Pade are significant by taking a slab geometry and a rela-47

tively large radius. Moreover, the introduction of the gy-48

roaverage operation tends to diminish the growth rate of49

the instability and this is amplified, when considering the50

direct (right) gyroaverage operator, instead of the Pade51

approximation. Linear analysis predicts a similar behav-52

ior, when we compare Pade approximation and the J053

Bessel function for kρ < 1 but a decrease of the linear 54

growth rate when 1 < kρ < 5. Note that the result re- 55

mains at the qualitative level, as here, in polar geometry, 56

the multiplication by the J0 Bessel function is not the ex- 57

act solution (expect for Fourier-Bessel functions), and this 58

differs from the cartesian geometry. Non-linear long-time 59

simulations should also be performed in the future to see 60

the impact on large wave number behavior. 61

Appendix: Derivation of the dispersion relation 62

In order to validate the linear part of the numerical results,we compute the dispersion relation with the gyroaverage.We make the following expansions:

f = f0 + εf1 + O(ε2), φ = φ0 + εφ1 + O(ε2)

with 63

f0(r, v) = feq(r, v) =n0(r) exp

(− v2

2Ti(r)

)

(2πTi(r))1/2, φ0 = 0. 64

Then we obtain

J√2μ(f) = J√

2μ(f0 + εf1) + O(ε2),

andJ√

2μ(φ) = J√2μ(φ0 + εφ1) + O(ε2).

Substituting the above relations into (10), we obtain 65

∂tf1−∂θJ√

2μ(φ1)r

∂rf0+v∂zf1−∂zJ√2μ(φ1)∂vf0 = O(ε). 66

Similarly, the equation 67

68

−(

∂2rφ +

(1r

+∂rn0(r)n0(r)

)∂rφ +

1r2

∂2θφ

)69

+1

Te(r)(φ − λ〈φ〉) =

1n0(r)

J√2μ

(∫f − feqdv

)70

71

becomes 72

−(

∂2rφ1 +

(1r

+∂rn0(r)n0(r)

)∂rφ1 +

1r2

∂2θφ1

)

+1

Te(r)(φ1 − λ〈φ1〉) =

1n0(r)

J√2μ

(∫f1dv

)+ O(ε).

(A.1)

We assume that the solutions have the form: 73

f1 = fm,n,ω(r, v)ei(mθ+kz−ωt), φ1 = φm,n,ω(r)ei(mθ+kz−ωt)74

J√2μ(f1) = fm,n,ω(r, v)ei(mθ+kz−ωt), 75

J√2μ(φ1) = φm,n,ω(r)ei(mθ+kz−ωt)

76

Page 15 of 16

with k = 2πnL . Then, we obtain1

(−ω + kv)fm,n,ω =(m

r∂rf0 + k∂vf0

)φm,n,ω (A.2)2

and the relation (A.1) becomes3

4

−(

∂2rφm,n,ω +

(1r

+∂rn0(r)n0(r)

)∂rφm,n,ω − m2

r2φm,n,ω

)5

+1

Te(r)(φm,n,ω − λδ0

nφm,0,ω) =1

n0(r)

∫fm,n,ωdv.6

7

If we assume that m �= 0 and n �= 0, the last relation and8

equation (A.2) lead to:9

10

−(

∂2rφm,n,ω +

(1r

+∂rn0(r)n0(r)

)∂rφm,n,ω − m2

r2φm,n,ω

)11

+1

Te(r)φm,n,ω =

1n0(r)

φm,n,ω

∫fm,n,ω

fm,n,ω

mr ∂rf0 + k∂vf0

kv − ωdv12

13

and then14

15

−(

∂2rφm,n,ω

φm,n,ω+

(1r

+∂rn0(r)n0(r)

)∂rφm,n,ω

φm,n,ω− m2

r2

)16

+1

Te(r)=

1n0(r)

φm,n,ω

φm,n,ω

∫fm,n,ω

fm,n,ω

mr ∂rf0 + k∂vf0

kv − ωdv.17

18

We make the approximations:

Φm,n,ω

Φm,n,ω≈ J0(κ

√2μ),

fm,n,ω

fm,n,ω≈ J0(κ

√2μ)

where κ ∈ R+. Rigorously the previous approximations

are true when f and Φ are Fourier-Bessel functions in(r, θ), i.e. when

fm,n,ω(r, v) = Jm(κr) × g(v)

and Φm,n,ω(r) = Jm(κr) (see Proposition in Sect. 3). In19

general, we are not in this case and this will explain the20

differences we observe in Figure 9. Then, by considering21

the previous approximations, we obtain:22

23

−(

∂2rφm,n,ω

φm,n,ω+

(1r

+∂rn0(r)n0(r)

)∂rφm,n,ω

φm,n,ω− m2

r2

)24

+1

Te(r)= J0(κ

√2μ)2

1n0(r)

∫ mr ∂rf0 + k∂vf0

kv − ωdv.25

26

By setting

I =∫ m

r ∂rf0 + k∂vf0

kv − ωdv

and by using the expression of f0, we have

I =∫ − v

Ti+ m

kr (∂rn0n0

− ∂rTi

2Ti+ v2∂rTi

2T 2i

)

v − ωk

f0dv.

Now, we introduce for n ∈ N:

In =1n0

∫vn f0

v − ωk

f0dv

and we obtain the relations:

I1 = 1 +ω

kI0, I2 =

ω

k

(1 +

ω

kI0

).

By using the change of variable v = (2Ti(r))1/2w and the 27

expression of f0, we have by setting k∗ = (2Ti)1/2k. 28

I0 =∫ exp

(− v2

2Ti

)

(2πTi)1/2(v − ωk )dv

29

=∫

exp(−ω)π1/2((2Ti(r))1/2w − ω

k )dv30

=1

(2Ti)1/2Z

( ω

k∗)

31

with 32

Z(z) =1√π

∫exp(−x2)

x−zdx = i

√π exp(−z2)(1−erf(−iz)), 33

erf(x) =2√π

∫ x

0

exp(−t2)dt. 34

Finally, for z = ω/k∗: 35

J0(κ√

2μ)2

n0(r)I = − 1

Ti(1 + zZ(z))

+m

k∗r

(Z(z)

(∂rn0

n0− ∂rTi

2Ti

)

+ z(1 + zZ(z))∂rTi

Ti

). (A.3)

References 36

1. N. Crouseilles, M. Mehrenberger, H. Sellama, CiCP 8, 484 37

(2010) 38

2. A. Dimits et al., Phys. Plasmas 7, 969 (2000) 39

3. A. Brizard, J. Plasma Phys. 41, 541 (1989) 40

4. T.S. Hahm, Phys. Fluids 31, 2670 (1988) 41

5. Z. Lin, W.W. Lee, Phys. Rev. E 52 5646 (1995) 42

6. V. Grandgirard, M. Brunetti, P. Bertrand, N. Besse, X. 43

Garbet, P. Ghendrih, G. Manfredi, Y. Sarazin, O. Sauter, 44

E. Sonnendrucker, J. Vaclavik, L. Villard, J. Comput. 45

Phys. 217, 395 (2006) 46

7. V. Grandgirard, Y. Sarazin, X. Garbet, G. Dif-Pradalier, 47

Ph. Ghendrih, N. Crouseilles, G. Latu, E. Sonnendrucker, 48

N. Besse, P. Bertrand, Commun. Nonlinear Sci. Numer. 49

Simul. 13, 81 (2008) 50

8. T. Gorler, X. Lapillonne, S. Brunner, T. Dannert, F. 51

Jenko, F. Merz, D. Told, J. Comput. Phys. 230, 7053 52

(2011) 53

9. T. Gorler, Multiscale Effects in Plasma Microturbulence, 54

Ph.D., Ulm, 2009 55

10. W.W. Lee, Phys. Fluids 26 556 (1983) 56

Page 16 of 16

11. R. Hatzky, T.M. Tran, A. Konies, R. Kleiber, S.J. Allfrey,1

Phys. Plasmas 9, 898 (2002)2

12. Y. Idomura, S. Tokuda, Y. Kishimoto, M. Wakatani, Nucl.3

Fusion 41 437 (2001)4

13. S. Jolliet, A. Bottino, P. Angelino, R. Hatzky, T.M. Tran,5

B.F. Mcmillan, O. Sauter, K. Appert, Y. Idomura, L.6

Villard, Comput. Phys. Commun. 177, 409 (2007)7

14. Y. Sarazin, V. Grandgirard, E. Fleurence, X. Garbet, Ph.8

Ghendrih, P. Bertrand, G. Depret, Plasma Phys. Control.9

Fusion 47, 1817 (2005)10

15. M. Mehrenberger, C. Steiner, L. Marradi, N. Crouseilles,11

E. Sonnendrucker, B. Afeyan, Vlasov on GPU, ESAIM12

Proc., 201313

16. M. Kreh, Bessel Functions, www.math.psu.edu/papikian/14

Kreh.pdf15

17. M. Abramowitz, I.A. Stegun, Handbook of Mathematical 16

Functions (Dover Publications, New York, 1965) 17

18. X. Garbet, Y. Idomura, L. Villard, T.H. Watanabe, 18

Gyrokinetic simulations of turbulent transport, Nucl. 19

Fusion 50, 043002 (2010) 20

19. SELALIB, http://selalib.gforge.inria.fr/ 21

20. R. Klein E. Gravier, P. Morel, N. Besse, P. Bertrand, Phys. 22

Plasmas 16, 082106 (2009) 23

21. N. Crouseilles, P. Glanc, S.A. Hirstoaga, E. Madaule, M. 24

Mehrenberger, J. Petri, Semi-Lagrangian simulations on 25

polar grids: from diocotron instability to ITG turbulence, 26

submitted 27

22. D. Coulette, N. Besse, JCP 248, 1 (2013). 28

23. V. Grandgirard et al., Commun. Nonlinear Sci. Numer. 29

Simul. 13, 81 (2008) 30