USC Progress Report: 3D EMPIC Simulations of Whistler Turbulence

I. Modeling Nonlinear Evolution of Whistler Turbulence: Local Simulation

II. Modeling Whistler Wave/Turbulence Propagation: Toward a Global Simulation Model

Joseph Wang, Ouliang ChangDepartment of Astronautical Engineering

University of Southern California

Acknowledgement:S. Peter Gary, Los Alamos National Lab

Guru Ganguli, Naval Research Lab

Summary of Recent WorkSummary of Recent Work

• Extension of previous study on ring-beam induced instabilities • Cross comparison/validation: EMPIC simulations of lower hybrid

wave/instability vs. Winske’s hybrid ES simulation

• 3D EMPIC simulations of whistler turbulence evolution

• Development of a global EMPIC simulation of whistler wave/turbulence propagation

This talk

I. I. Modeling Nonlinear Evolution of Whistler Turbulence: Local Simulation

IntroductionIntroduction• Two recent studies on the evolution of whistler turbulence:

• Saito et al (2008), Gary et al (2010):

• 2D EMPIC simulation showed the evolution is dominated by forward cascade from long to short wavelengths

• Ganguli et al (2010):

• Whistler turbulence is fundamentally a 3-D phenomena

• Pseudo-3D EMPIC simulation which includes the

effect showed that the evolution is dominated by inverse cascade from short to long wavelengths.

2-D simulations may not allow the development of important nonlinear wave-wave processes

0)( Bnk

Objective:

To perform 3D EMPIC simulations of whistler turbulence and to investigate the “forward cascade vs inverse cascade” issue in a fully 3D setup

Previous Work [Ganguli et al., 2010]Previous Work [Ganguli et al., 2010]pseudo-3D; pseudo-3D; whistler instability launched by a ring beamwhistler instability launched by a ring beam

X

Y

Z

oB

config.) (Saito0θ al. et

oB

06θ

. 0θ 06θ

Ongoing WorkOngoing Workfully 3D; fully 3D; whistler instability launched by a prescribed spectrum of whistlerswhistler instability launched by a prescribed spectrum of whistlers

z (Bo, ||)

xy

• Initial simulations focus on 3D vs. 2D using the Satio-Gary initial condition

• Larger simulations to determine whether the forward cascade or the inverse cascade is going to “win” are currently running

Due to computer limitations: this talk addresses the intermediate questions:1) How does the 3D forward cascade compare against the 2D forward cascade 2) Is there an inverse cascade in a fully 3D setup of Saito-Gary initial condition

Computational Challenge to Perform a Fully 3D EMPIC Computational Challenge to Perform a Fully 3D EMPIC Simulation of Whistler TurbulenceSimulation of Whistler Turbulence

• Simulation Parameters to extend Saito et al (2008) 2D setup to full 3DMemory Requirement Estimation: 8TBParticle per Cell: ~64 particles per cell per speciesCell resolution: dx= dy=1 Debye length=0.1c/omega_peDomain size: 1024X1024X1024>109

Total # of particles: >128X109

Computing Time Estimation on USC HPC: >35 days on 256 nodes dt=0.05*(omega_pe) -1 CPU Time/particle/step: ~2.75E-7sTotal CPU time for 20000 steps (447Ωce

-1): : >8770 day

• USC High Performance Computing Center Parallel Supercomputer:typical node: quad-core/dual processortypical memory each node: 12-16GBtheoretical peak speed: 83.29teraflops on 1460nodesprocessor availability: <256 node for each run

Parallel Simulation Code DevelopmentParallel Simulation Code Development

• Code development, optimization, and validation completed• Implementation using hybrid MPI+OpenMP

• New particle sorting algorithm to enhance data locality and speed up the computation

• Various initial loading of whistler waves for pre-conditioned simulation

• Code Validation against pervious simulations

Simulation SetupSimulation Setup

• Initial condition:

• Initial waves:

where

• 2D configuration: 42 models calculated from S. Peter Gary’s dispersion solver

On a 2D grid containing the background magnetic field, lay out 42 modes with k_|| and k_perp calculated from dispersion solver.Put the appropriate amount of magnetic energy in each component for each mode based on dispersion solver. Each mode should have the same total magnetic energy density, by assumption.Use Faraday's Law to get the appropriate amount of electric field energy in each component for each mode.Use Ampere's Law to compute the components of the fluctuating electron velocities for each mode, which is imposed as a perturbation of the average velocity of the electron velocity distribution.

• 3D configuration: 84 modes rotate the plane containing the wave vectors 90 degrees about Bo, and repeat the

above process

Simulation CasesSimulation Cases

• Case 0: code validation. based on Saito et al (2008)

• Physical parameters: 2D 102.4 λe, ωpe/Ωce=2.236, βe=0.1, Te=Tp, c/vte=10

• Cell=1024X1024; 64particle/cell/species; total particle=134.2million

• Case 1: to compare 3D forward cascade vs. 2D forward cascade

• Parameters: 51.2 λe, ωpe/Ωce=0.707, βe=0.01, Te=Tp, c/vte=10

• 1a 2D:Cell=512X512; total particle=33million

• 1b 3D: Cell=512X512X512; 32particle/cell/species; total particle=8.59E9

• Case 2: to determine whether inverse cascade exists in the Saito-Gary setup

• Parameters: 51.2 λe, ωpe/Ωce=0.707, βe=0.01, Te=Tp, c/vte=10

• 2a: 2D. Cell=512X512; total particle=33million

• 2b: 3D. Cell=512X512X512; 32particle/cell/species; total particle=8.59E9

• Case 3: forward cascade vs. inverse cascade (ongoing)

• Parameters: 102.4 λe, ωpe/Ωce=0.707, βe=0.01, Te=Tp, c/vte=10

• 3D. Cell=1024X1024X1024; 64particle/cell/species; total particle=137E9

Mi/Me=1836

Simulation 0: Code ValidationSimulation 0: Code Validation

Gary et al 2008

Ωce*t=2011

Saito et al 2008

Ωce*t=447

K||λe

Kyλe

Log(δB^2/B0^2)

Simulation 1: 2DSimulation 1: 2D

Ωce*t=1414Ωce*t=0Log(δB^2/B0^2)

)(

)(tan

,

,

yzxk

yzykB kkBk

kkBk

δ

δθ

2

222

Case 1a 2D:Case 1a 2D:

Ωce*t=1414Ωce*t=0 Ωce*t=1414Ωce*t=0

Simulation 1: 3DSimulation 1: 3D

Case 1b 3D:Case 1b 3D:

Y-Z Plane X-Z Plane X-Y Plane

at Ωce*t=141.4

312 .tan Bθ 112 .tan Bθ 212 .tan Bθ

Log(δB^2/B0^2)

)( xkB2δ )( ykB2δ )( zkB2δ

Ωce*t=0Log(δB^2/B0^2)

)(

)(tan

,

,

yzxk

yzykB kkBk

kkBk

δ

δθ

2

222

Case 2a 2D:Case 2a 2D:

Ωce*t=1414Ωce*t=0

Simulation 2: 2DSimulation 2: 2D

Ωce*t=141.4

Ωce*t=141.4Ωce*t=0Log(δB^2/B0^2)

Case 2b 3D:Case 2b 3D:

Simulation 2: 3DSimulation 2: 3D

both forward cascade and inverse cascade seem to exist in the 3D Saito-Gary setup; however, which process is going to “win” under realistic magnetospheric parameters remain to be answered

II. II. Modeling Whistler Wave/Turbulence: Toward Global Simulation

IntroductionIntroduction

• All simulation studies on whistler turbulence so far are based on local simulations with periodic BC

• Local simulations do not allow the study of global characteristics of whistler turbulence associated with whistler emission and propagation

Objective:

• To develop a global simulation model for studying anomalous absorption of whistler waves injected by a transmitter

This talk:

• Development and testing of an EMPIC code with absorbing BC for whistler waves

Algorithm: Wave Absorption at Boundary

• Effective damping region

• Outgoing waves at simulation domain boundary are absorbed using a damping region scheme (Umeda, Comp. Phys. Comm., 2001)

• In the damping region, an amplitude damping factor is used to gradually reduce the amplitude (energy) of outgoing waves at each time step and a phase retarding factor is used to gradually reduce the propagation speed of outgoing waves (and plasmas).

Code Testing: 2D Perturbations in Maxwellian Plasma

Initial Condition:

2D sinusoidal wave propagating in X direction. Ey(i,j,k)=Bo*sin(re/20*2*pi); Bz(i,j,k)=Bo*sin(rm/20*2*pi)

re, rm are distance to center point of 2D domain plainwave length: 20cell

Background Plasma:•Background magnetic field in the Z direction•Speed of light c =8•ve_th=1 vi_th=0.125 v_d=0

Domain and Wave Damping Region:

•damping region: 40 cell; Domain size: 280X280X1;

Boundary Condition•Field: Wave absorption•Particle: injection and absorption•Injection: one-sided flux of background distribution

2D Perturbations in Maxwellian Plasma

Bz animation) Oscillating Bz

• 2D Perturbation propagation in X direction

Initial Simulation Results:Whistler Propagation in Open Domain

• 2D open (absorbing) boundary condition, cell size 300*1*512• Open BC in X and Z, periodic BC in Y• Absorbing region size: 100 cells in each side• Background B in Z direction (in simulation plain)

Y

Z

X

Bo

100

100

100

100 100312

Simulation Parameters

Whistler Wave

• Initial whistler mode loaded in X-Y direction

• Dispersion relation

• Choose k , assuming

22

2

kcce

pe ωω

10006040

40

λ

ωωω

ˆ,.ˆ/.ˆ

.

ck

cck pepe

ce

pece ωω 18040 ..

otkzj

ytkzj

x

tkzjy

tkzjx

BEeE

BeE

jB

ejEEeEE

101010

111

11

/./.,)()(

)()(

ωω

ωω

ηη

Whistler Propagation In Vacuum (No Plasma)

Propagation of Bx Field Energy Comparison

Totoal time: t*omega_pe=40Time interval: dt*omega_pe=0.8

Whistler Propagation In Vacuum (No Plasma)

Energy Frequency Spectrum (Bx)

Energy Frequency Spectrum (Ex)

Energy Wavevector Spectrum (Bx)

Consistent with initial loaded fluctuation

Wave number spectrum at t*omega_pe=20

Totoal time: t*omega_pe=40

By

Field Energy Comparison

ExTotoal time: t*omega_pe=120Time interval: dt*omega_pe=0.8

Energy Frequency Spectrum (By)

120

Energy Frequency Spectrum (Ex)

Energy Wavevector Spectrum (By)

Whistler modePlasma mode

Consistent with initial loaded fluctuation

• Magnetic field frequency and wavenumber match the whistler dispersion relation.

• Electric field is dominated by plasma mode.

Wave number spectrum taken at t*omega_pe=50

Totoal time: t*omega_pe=120

By

Field Energy Comparison

Ex

Totoal time: t*omega_pe=200Time interval: dt*omega_pe=0.8

Energy Frequency Spectrum (By)

200

Energy Frequency Spectrum (Ex)

Energy Wavevector Spectrum (By)

Whistler mode

Plasma mode

Consistent with initial loaded fluctuation

• Magnetic field frequency and wavenumber match the whistler dispersion relation.

• Electric field shows both whistler mode and plasma mode. Totoal time: t*omega_pe=200

Wave number spectrum at t*omega_pe=50

By

Field Energy Comparison

Ex

Magnetic and electric field propagation become turbulent in later time.

Totoal time: t*omega_pe=150Time interval: dt*omega_pe=0.8

Energy Frequency Spectrum (By)

150

Energy Frequency Spectrum (Ex)

Energy Wavevector Spectrum (By)

Whistler mode

Plasma mode

Consistent with initial loaded fluctuation

Whistler mode

• Magnetic field frequency and wavenumber match the whistler dispersion relation.

• Electric field shows that whistler mode is dominating. Totoal time: t*omega_pe=150

Wave number spectrum at t*omega_pe=50

Summary and Conclusions

• local simulation:– new parallel empic code has been implemented and optimized for

3D simulations of whistler turbulence evolution– First fully 3D EMPIC simulation of whistler turbulence carried out– Initial results (run time: days) showed both forward and inverse

cascade using the Saito-Gary initial condition– Larger scale simulations using more than 109 cells and 1011 particles

(run time: weeks) to resolve the “forward cascade vs. Inverse cascade” issue are ongoing

• global simulation:– new subroutines for wave absorption are developed and tested for

simulation of whistler turbulence associated with whistler emission from a transmitter in open space

– Future work will develop a whistler emission model and study whistler turbulence evolution within the context of emission/ propagation

Simulation Case 1aSimulation Case 1a

Case 1a 2D:Case 1a 2D:

• Dash line: (f(v,t) – f(v,t=0))/f(0,t=0)

Simulation Case 2aSimulation Case 2a

Case 2a 2D:Case 2a 2D:

• Dash line: (f(v,t) – f(v,t=0))/f(0,t=0)