a particle simulation with accelerated gyrokinetic electron and fully-kinetic ion (gefi) code

A Particle Simulation With Accelerated Gyrokinetic Electron and Fully-kinetic Ion

(GeFi) CodeSpeaker:Wei Kong

(孔伟 )

Nankai university

Xueyi Wang1, Yu Lin1, Liu Chen2 , Huasheng Xie2 and Peng Wang3

1: Auburn university2: Zhejiang university

3: NVIDIA

Hangzhou, April 2012

ContentsContents

Introduction to GeFi Accelerated GeFi with CUDA electron beam instability in the center of Harris current

sheet

GeFi Simulation 2

Introduction to GeFi Introduction to GeFi Motivation

Important physics of kinetic process involves both electron and ion scales. (collisionless magnetic reconnection ranges from short electron scale to global Alfven scale).

Fully particle simulation often employ artificial mass ratio in order to accommodate limited computing resources. also suffers the numerically unstable due to rather more steps.

Hybrid simulation based on fully-ion & fluid-electron mainly solve problems related with only ion dynamics. Hybrid simulation based on fully-ion and drift kinetic-electron has a salient drawback that drops the important electron gyro-radius & polarization effects.

GeFi particle simulation aims at including both electron and ion kinetics, meantime computing efficiently. (First published by Yu Lin, Xueyi Wang, Zhihong Lin and Liu Chen at 2005)

GeFi Simulation 3

Introduction to GeFi Introduction to GeFi

ce

k k

GeFi Simulation 4

Advantages of GeFi simulation

Orders of magnitude improvements in time step and grid space are achieved by removing the rapid cyclotron electron-motion with Larmor radius retained, such that the super-computer can be greatly utilized to study a global problem such as collisionless magnetic reconnection.

Problems with disparate temporal and spatial scales (modes ranging from magnetohydrodynamic wave to kinetic wave) can be solved at an equal footing.

What GeFi could resolve:

Frequency with due to the gyro-averaging of electron motion.

Wave number with due to eliminate the high-frequency Langmuir oscillation along magnetic field.

Introduction to GeFi Introduction to GeFi GeFi kernals —— ion equations of motion

To keep symmetry in driving ions and particles:

then have

where and is the background magnetic field, is the perturbed scalar potential, and is the perturbed vector potential.

2

/

1/ 2 | / |

p mv qA c

H m p eA c e

( / )( / ) / i

p q v A cx p q A c m

A A A A A

GeFi Simulation 5

Introduction to GeFi Introduction to GeFi GeFi kernals —— electron equations of motion

Employ gyro-kinetic ordering for electrons

where and <…> indicates the gyro-averaging.

1

ee

ce

e

BkL B

k

* *

* *

[ ]

[ ]

p b q B

cR v b b q BqB

GeFi Simulation 6

* *( / ) ( ) , /e ceb b v b b b v A c

Introduction to GeFi Introduction to GeFi GeFi kernals —— potential equations (1) generalized Poisson’s equation

Ampere’s law

electron force balance equation

and were intentionally placed at left-hand side or right-hand side to calculate conveniently.

2 2*

2 * 2

2

* 2

4[(1 ) ] ( )(1 )

(4 )(1 )

pe e ee

ce e

e ep

e

n m cB

n qR HB

2 22

2 2

4( )( ) ( )pi peA A b b J Jc c c

2 2

2 * 2

2*

2

(4 )1[(1 ) ](1 )

1[(1 ) ( )]

pe e e

ce e e e

pep w B B

ce e e

n qn q B

R H H S Hn q

GeFi Simulation 7

Introduction to GeFi Introduction to GeFi

4 [ ]p i i i eR q n q N

( ) ,e e i iJ p F dp p f dp b

[ / ]B eg iS P J B c

2 2

44 [ ( ) ( )]e ep e i

n nH q J A A BB B

2*

2 *

1 [ ]1

pew e

ce e

H

* * *1 4[ ( ) ( ) ]4B e e B

JH A B A Hc

21 { [( ) ] / 2}4BH B B B

( )4cJ B b

,eg e e e eP m p p F dp

GeFi kernals —— potential equations (2)

* / 2e e

GeFi Simulation 8

* 2*(1 ) ( )

4 4e e

ee

B m cBq

Accelerated GeFi with CUDA Accelerated GeFi with CUDA

GeFi & CUDA A typical simulation with 256x64 grids, 100 particles per cell

and 4000 time steps (call 32 cores) will cost about 3 hours . For the 3D global simulation, much more longer. Is there a way to run faster?

Yes, with CUDA!

Get to know CUDA CUDA refers to a parallel computing architecture, which

mainly includes ISA instruction set(PTX) and hardware for graphics and computing(NVIDIA GPU). Through “the most-intensive computing” in GPU.

GeFi Simulation 9


GeFi Simulation 10

(1) Diagnose to find which should be optimized:

……………….


GeFi Simulation 11

(2) Redefine variables and functions in .cu file, as soon as possible avoid calling the outer

__forceinline__ __device__ void A_CROSS_B(float *A, float *B, float *C, float &ABSC…){ C[0]=A[1]*B[2]-A[2]*B[1]; C[1]=A[2]*B[0]-A[0]*B[2]; C[2]=A[0]*B[1]-A[1]*B[0]; ABSC = sqrt(powf(C[0],2)+powf(C[1],2)+powf(C[2],2));}

__global__ void acce_gpu_loop_kernel(float *QVE, float *QVE0, float *VGCE, int NE, float XMIN, float YMIN, float ZMIN)

Just copy source code!! ……


GeFi Simulation 12

(3) At last wrapper the kernal to run at GPU, then transfer back to CPU

extern "C"{ void acce_gpu_loop_(float *QVE, float *QVE0, float *VGCE, int *NE, float *XMIN, float *YMIN, float *ZMIN, float *DX, float *DY, float *DZ, int *I0A, int *J0A, int *K0A, ……..}

souce code …CALL acce_gpu_loop( QVE, QVE0, VGCE, NE, XMIN, YMIN, ZMIN, DX, DY, DZ, I0A, J0A, K0A, I1A, J1A, K1A, NX, NY, NZ, BTOT_ALL, B_TOT_BAR …….)source code …


Performance of upgraded GeFi

• Initial port of acce: ~ 30x– (NXT,NYT,NZT)=(1,65,257)– CPU acce time: 14 sec/step (PGI Fortran compiler, -O2)– GPU acce time: 0.47 sec/step (kernel: 0.15 sec)

optimizated by Dr. P. Wang (NVIDIA)

• Second port of accp: ~ up to 45x– (NXT,NYT,NZT)=(1,65,257)– CPU acce time: 14 sec/step (PGI Fortran compiler, -O2)– GPU acce time: 0.30 sec/step

GeFi Simulation 13

Electron beam plasmaElectron beam plasma Beam instability Exists in the laboratory and space plasma, may contributes to

the fast magnetic reconnection. Variations: cold and warm , weak and strong, isotropic and

anisotropic, linear and nonlinear… We more concerns about the ion beam plasma due to an

instability localized at the center of Harris current sheet.(Wang PoP et al., 2008).

GeFi Simulation 14

Electron beam plasma Electron beam plasma

GeFi Simulation 15

For benchmark, we begin with an electron beam plasma:

Cold dispersion relation(cold ion and cold electron beam, Verdon et al., PoP, 2011):

While our GeFi simulation includes the ion and electron thermal effects.

22 2 2 2 22 2

2 2 4 2 2 2 2 2

22 22

22 2 2

1 1 1

cos 0,

pe pe pe d

e e

pee e

i e e i

k vk c k c k c

m mm m k c

/ / dk v


GeFi Simulation 16

Design the simulation: Y

B

beam X wave


GeFi Simulation 17

Electron beam FK theory(line) & cold theory(line) & GeFi simulation(contour) GeFi simulation(circle)

23, 0.04, 0, 1.553 (88.98 )pei e de B wave

ce

V rad

0deV


GeFi Simulation 18

Weak electron beam

(1) Consistent with higher- ranch (2) zero frequency branch,not grows. (up to 10 )

23, 0.04,

1.553 (88.98 )

pei e

ce

B wave rad

1i

70de iV V


GeFi Simulation 19

Strong electron beam

(1) Forward propagating(2) Lower growth rate

Resonance with ion-Cyclotron motion?

23, 0.04,

1.553 (88.98 )

pei e

ce

B wave rad

300de iV V


GeFi Simulation 20

More strong electron beam (Verdon et al. PoP, 2011 )

For such a distribution, Thermal nearly not play effects?

23, 0.04,

1.553 (88.98 )

pei e

ce

B wave rad

500de iV V


GeFi Simulation 21

GeFi simulation (nonlinear)


GeFi Simulation 22

Analyze(1):By , Bz, fluctuates at the same level. Ex >> Ey and Ez

Note: the main magnetic field is designed to Y direction.


GeFi Simulation 23

Analyze(2):EM or ESES ?

610E

910E


GeFi Simulation 24

Analyze(3):Resonance?

Which gives , so A large gap between and , even considers the width

of electron velocity distribution (300 162) Anyway ,next we show the phase condition of electrons and

ions(in the frame of moving wave with ).

23, 0.04, 300 , 1.553 (88.98 )

3.51, 3.75

pei e de i B wave

ce

total i ci

V V rad

k

0.1331ik 60p iV V

pV deV

60p iV V

iV


GeFi Simulation 25

Analyze(4):eles phase condition(at the linear stage)

1 10.268 1.36i i


GeFi Simulation 26

Analyze(5):eles phase condition(at the nonlineaer stage)

1 12.4 3.5i i


GeFi Simulation 27

Analyze(6):Ions phase condition(at the linear stage)

1 10.268 1.36i i


GeFi Simulation 28

Analyze(7):Ions phase condition(at the nonlineaer stage)

1 12.4 3.5i i

GeFi Simulation 29

Seed :strong electron beams(with thermal effects) (check the polarization) generates perturbed Ex(dominant)

acclerates ion (Vx) resonance with wave along x

ConclusionsConclusions

GeFi Simulation 30

A CUDA-version GeFi pic code was given. We studied the weak and strong electron beam plasma

with GeFi simulation, and compared with the cold beam theory. One typical nonlinear case with considering particles thermal effects and strong electron beam was analyzed briefly.

Basically, Our GeFi simulation shows that the thermal effects of the electron beam plays an important influence on the plasma instability.

To be continued…

GeFi Simulation 31

Thanks!

To be continuedTo be continued

GeFi Simulation 32

and V. S. Background effects Tp/Te effects Beta effects Anisotropic effects Saturation level Boundary effects To benchmark the weak-beam nonlinear behavior with the results of Kainer

et al. (S. Kainer, J. Dawson, R. Shanny and T. Coffey, Phys. Fluids 15 (1972) 493.).

Ion beam plasma Beam plasma in Harris current sheet.

B wave maxdeV

a particle simulation with accelerated gyrokinetic electron and fully-kinetic ion (gefi) code

Documents

gefi gefi simulation

gefi accelerated gefi

cuda gefi simulation

gefi particle simulation

cuda gefi cuda

kinetic ion gefi codespeaker

hybrid simulation

typical simulation