gysela: turbulent transport in tokamak plasmas - genci
Post on 21-Apr-2022
5 Views
Preview:
TRANSCRIPT
GYSELA: Turbulent transportin tokamak plasmas
V. Grandgirard1, G. Latu1
Collaborations with physicists:J. Abiteboul2, Y. Dong3, D. Esteve1, X. Garbet1,J.B Girardo1, Ph. Ghendrih1, F. Palermo1,Y. Sarazin1, A. Strugarek7, D. Zarzoso2
Collaborations with mathematicians:A. Back4, T. Cartier-Michaud1, M. Mehrenberger5,L. Mendoza2, E. Sonnendrucker2
Collaborations with computer scientists:J. Bigot6, C. Passeron1,F. Rozar1,6, O. Thomine1
1CEA, IRFM, Cadarache, France2IPP Garching, Germany3LPP, Paris, France4CPT, Marseille, France5IRMA, Strasbourg, France6Maison de la Simulation, Saclay, France7Montreal university, Canada
ANR GYPSI - ANR G8-Exascale NufuseADT-INRIA SELALIB - AEN-INRIA Fusion
, Virginie G N GENCI Gd Challenges N 21 May 2013 1
Turbulence governs Fusion plasma performance
Magnetic ConfinementIn a plasma, particles follow magnetic field lines: Particles may be confined in a toroidal magnetic field
In the sun, plasma is confined by gravityIn a tokamak, plasma is confined by a magnetic field
magnetic toroidal geometry (r , θ, ϕ)
I Certainty: Turbulence limit the maximal value reachable for n and T
à Generate loss of heat and particlesà Confinement properties of the magnetic configuration
I Subject of utmost importance à optimizing experiments like ITER andfuture reactors.
, Virginie G N GENCI Gd Challenges N 21 May 2013 2
Complex interplay between fields and particles
I Charged particle motion governed by electromagnetic fieldsI Electromagnetic fields governed by charge density ρ and current j
à self-consistent treatment required
à
I Plasma response: The most accurate⇒ Gyrokinetic
, Virginie G N GENCI Gd Challenges N 21 May 2013 3
Kinetic model for plasma turbulence
I Fields à Maxwell’s equationsI Electrostatic (B = const): E = −∇φ (φ electrostatic potential)I “large scale” (> λDebye)
à Quasi-neutrality equation:
ρ(x, t) =∑
s
nsqs = 0 with ns =
∫fsdv
I Particles à Kinetic approach mandatoryI Fusion plasmas weakly collisional⇒ far from the fluid state
à Boltzmann equation:
∂fs∂t
+ v ·∂fs∂x
+dvdt·∂fs∂v
= C(fs) + S
6D function of s specie fs(x,v) (3D in space and 3D in velocity)
, Virginie G N GENCI Gd Challenges N 21 May 2013 4
Gyrokinetic theory:à large phase space reduction 6D to 5D
Kinetic theory: à 6D distribution function of particles
I Fusion plasma turbulence is low frequency:ωturb ∼ 105s−1
ωci ∼ 108s−1
I Phase space reduction: fast gyro-motion is averaged outà Adiabatic invariant: magnetic momentum µ = msv2
⊥/(2B)
à Velocity drifts of guiding centers
CEMRACS 2010, Marseille
Transverse driftsTransverse drifts Transverse & parallel dynamics:
Projection on the transverse plane ( ):
(with )
electric drift curvature + ∇B drifts
vG//
vG⊥B, Large reduction memory/CPU time
/ Complexity of the system
Gyrokinetic theory: à 5D distribution function of particlesf(r , θ, ϕ, v‖, µ)
, Virginie G N GENCI Gd Challenges N 21 May 2013 5
Gyrokinetic codes require state-of-the-art HPC
I GK codes require state-of-the-art HPC techniques and must runefficiently on more than thousands processors.
I non-linear 5D simulationsI multi-scale problem in space and time
I time: ∆t ≈ γ−1∼ 10−6s → tsimul ≈ few τE ∼ 10s
I space: ρi → machine size a ρ∗ ≡ρi
a 1
3 ρ∗ ITER = 1/512
3 Number grid points ∼ (ρ∗)−3
à
Huge mesh for global simulations
, Virginie G N GENCI Gd Challenges N 21 May 2013 6
GK codes: a highly competitive activity
GYSELA characteristics:
I global : capture large scale events (at the opposite of local codes)I full-f : no scale separation between equilibrium and perturbation as δfI Semi-Lagrangian scheme (= mixed between PIC and eulerian approach)
, Virginie G N GENCI Gd Challenges N 21 May 2013 7
GYSELA: collaboration with global codes
GYSELA characteristics:
I global : capture large scale events (at the opposite of local codes)I full-f : no scale separation between equilibrium and perturbation as δfI Semi-Lagrangian scheme (= mixed between PIC and eulerian approach)
, Virginie G N GENCI Gd Challenges N 21 May 2013 7
GYSELA deployed on 9 HPC machines
I GYSELA is actually present on 9 different HPC machines.→ Deployments in 2012-2013: CURIE, HELIOS, CCAMU, Poincare
→ Code adaptation for BlueGene architecture (2012-2013): JUQUEEN, TURING
[Ch. Passeron, G. Latu, J. Bigot (Post-Doc MDS)]
Dedicated to development :- Local cluster : NORMA (256 cores) (2007)- « Mésocentres »: CCAMU-Marseille (~1000 cores) (2012) Maison De la Simulation-Saclay (~1500 cores) (2013) 512 cores dedicated to GYSELA (paid by ANR GYPSI)
Dedicated to development :- Local cluster : NORMA (256 cores) (2007)- « Mésocentres »: CCAMU-Marseille (~1000 cores) (2012) Maison De la Simulation-Saclay (~1500 cores) (2013) 512 cores dedicated to GYSELA (paid by ANR GYPSI)
Dedicated to production : - JADE (CINES Montpellier/France) : ~ 12.000 cores - HPC-FF (Juelich/Germany) : ~ 10.000 cores - TURING (IDRIS Orsay/France) : 65.536 cores
Dedicated to production : - JADE (CINES Montpellier/France) : ~ 12.000 cores - HPC-FF (Juelich/Germany) : ~ 10.000 cores - TURING (IDRIS Orsay/France) : 65.536 cores
Dedicated to production : - CURIE (Saclay/France) : ~ 80.000 cores - IFERC (Rokkasho/Japon) : ~ 80.000 cores - JUQUEEN (Juelich/Germany) : 458.752 cores
Dedicated to production : - CURIE (Saclay/France) : ~ 80.000 cores - IFERC (Rokkasho/Japon) : ~ 80.000 cores - JUQUEEN (Juelich/Germany) : 458.752 cores
HPC pyramid
I HPC-FF and IFERC dedicated to magnetic fusion community
, Virginie G N GENCI Gd Challenges N 21 May 2013 8
GYSELA CPU time needs à Exponential increase
ITER simulation8 192 cores during 1 month= 6.1 million h. CPU= 6 centuries of computation
~300 billions of points
ITER simulation8 192 cores during 1 month= 6.1 million h. CPU= 6 centuries of computation
~300 billions of points
1 century
2500 x
GYSELA CPU time consumption 25 millions h. CPU16384 cores
20 To of storage
25 millions h. CPU16384 cores
20 To of storage
I GYSELA CPU time needs increase exponentially
I Each Grand Challenge doubles the number of accessible hours per year
I 36 millions of hours for 2013 (PRACE,GENCI,IFERC)à More than 90% dedicated to production runs
: Grand Challenge TURING: 13 millions of hours
, Virginie G N GENCI Gd Challenges N 21 May 2013 9
GYSELA parallelisation improvements
I Possibility to access to the totality of the machine during “GrandChallenge” sessions give opportunity to test and improve parallelisationof the code
Relative efficiencyNumber of
coresWeak Scaling Strong Scaling
Gd Challenge CINES 2010
92 % 82 % 8192
Gd Challenge CURIE
(mars 2012)91 % 61 % 65536
Porting on Blue Gene Architecture => Communication schemes rewritten
Gd Challenge TURING
(janv 2013)92 % 61 % 65536
, Virginie G N GENCI Gd Challenges N 21 May 2013 10
WEAK scaling: (on JUQUEEN - Juelich)Relative efficiency of 91% on 458 752 cores
I Parallel communication schemes completely rewrittenI Tests performed on the totality of JUQUEEN/Blue Gene machine (Juelich)
64 128 192 256 320 384 448Nb. of Kcores (x 1000)
0
30
60
90
120
150
180Vlasov solverField solverDerivatives computationDiagnosticsTotal for one run
Execution time, one Gysela (Weak Scaling - Juqueen)
64 128 192 256 320 384 448Nb. of Kcores (x 1000)
0
20
40
60
80
100
120
Vlasov solverField solverDerivatives computationDiagnosticsTotal for one run
Relative efficiency, one run (Weak scaling - Juqueen)
I Weak scaling: Relative efficiency of 91% on 458 752 cores .
I PRACE preparatory access (April 2012 - Nov 2012): 250 000 hoursI ANR G8-Exascale via P. Gibbon.
, Virginie G N GENCI Gd Challenges N 21 May 2013 11
GYSELA: Biggest simulation ever run
Grand Challenge CINES 2010:
à A simulation close to ITER-size scenario (ρ∗ = 1/512) performed on 1/4torus with additional heating power of 60 MW during 1 ms
ion temperature fluctuationsin the turbulent saturated phase
3 A 5D mesh of 272 109 points(r , θ, ϕ, v‖, µ) = (1024 × 1024 × 128 × 128 × 16)
3 > 6.1 million hours monoproc.I ∼ 31 days on 8192 processors
à
3 6.5 TBytes of data to analyse
I 1.5 TBytes for 2D and 3D savingsI 5 TBytes for restart files
[J. Abiteboul EPS2010, Y. Sarazin IAEA2010]
, Virginie G N GENCI Gd Challenges N 21 May 2013 12
What’s new in physics
à What has been added since 2010 in terms of physicsI Generation & transport of toroidal rotation / Role of turbulence & boundary
conditionsI [J. Abiteboul et al., PPCF 2013]
I Transport barrier relaxations with Er shearI [A. Strugarek et al., PPCF 2013]I [A. Strugarek et al., PRL submitted]I [Y. Sarazin, V. Grandgirard and A. Strugarek, La Recherche, nov. 2012]
I Interaction energetic particles & turbulence-via EGAMsI [D. Zarzoso et al., PRL 2013]
I Comparison with experimentsI [invited G. Dif-Pradalier , TTF 2013]
I Adding of impurities à Expensive rewritten of the codeI Now one MPI world for speciesI Energy transfer operator & Moment transfer operatorI GC Curie 2012 à Source of impurities is mandatory
, Virginie G N GENCI Gd Challenges N 21 May 2013 13
What’s new in physics
à What has been added since 2010 in terms of physicsI Generation & transport of toroidal rotation / Role of turbulence & boundary
conditionsI [J. Abiteboul et al., PPCF 2013]
I Transport barrier relaxations with Er shearI [A. Strugarek et al., PPCF 2013]I [A. Strugarek et al., PRL submitted]I [Y. Sarazin, V. Grandgirard and A. Strugarek, La Recherche, nov. 2012]
I Interaction energetic particles & turbulence-via EGAMsI [D. Zarzoso et al., PRL 2013]
I Comparison with experimentsI [invited G. Dif-Pradalier , TTF 2013]
I Adding of impurities à Expensive rewritten of the codeI Now one MPI world for speciesI Energy transfer operator & Moment transfer operatorI GC Curie 2012 à Source of impurities is mandatory
, Virginie G N GENCI Gd Challenges N 21 May 2013 13
ITER-type simulation for a energy confinementtime unreacheable
I Compromise between machine size and simulation up to energy confinementtime must be found
I GYSELA simulation close to ITER-like parameters : 272 billions of pointsI Longest time simulation: 106/Ωc ∼ 1/2 confinement time
Number of
points(*=
i/a)
Time /cNumber of
cores
Number of days
of simulation
Gd ChallengeCINES 2010
272 billions(*=1/512)
147 840 8192 31
Gd ChallengeCURIE 2012
33 billions(*=1/150)
678 510 16384 15
=> Adding of Tritium 32768 4
Comparison with
experiment(in progress)
87 billions(*=1/512)
1 000 000 5520 23
, Virginie G N GENCI Gd Challenges N 21 May 2013 14
Complex interplay between Energetic particlesdriven modes (EGAMs) & turbulence
Grand Challenge CURIE 2012:
When SEP is on (time> 1800a/cs):
I Slow modification of Fs
I Phase A:Transient Transport Barrier inouter region ρ < 0.5
I Phase B:Saturated EGAMS interact withturbulence
à No regulation of turbulenceby EGAMs
[Zarzoso, PRL 2013 ; Sarazin, IAEA 2012]
, Virginie G N GENCI Gd Challenges N 21 May 2013 15
Transport barrier dynamics
Grand Challenge CURIE 2012:
I A vorticity source implemented in the codeI Transport Barrier triggered by prescribed E × B sheared flowI The barrier experiences quasi-periodic relaxation events
Snapshots of non-axisymmetricelectric potential fluctuations
[A. Strugarek, La Recherche (2013), PPCF (2012), submitted to PRL (2013)], Virginie G N GENCI Gd Challenges N 21 May 2013 16
GYSELA: On the road towards Exascale
I GYSELA is already using currently Petascale machine
→ Now ITER-like ion simulation: 272 109 points à 6 106 CPU/hours
I GYSELA will require Exascale machine for realistic kinetic electrons
→With electrons: ρions/ρelec = 60à mesh size ×603 and time step/60 !!!
, Virginie G N GENCI Gd Challenges N 21 May 2013 17
Conclusion - Perspectives
I Each GYSELA simulation = a numerical experiments→ Several weeks on several thousands of core
(ex: Grand Challenge Curie 2012: 15 days on 16384 cores)→ Several TBytes of data to store and to analyse
I Exascale HPC will be required for realistic simulation with both ions andkinetic electrons→ Promising results: Weak scaling - relative efficiency of 91% on 458 752 cores
I Work in progress for Exascale:→ Fault tolerance improvement [O. Thomine] (Post-Doc ANR-G8 Exascale)]
→ Memory reduction per nodes [F. Rozar] (PhD IRFM/ Maison Simulation)
→ Adapting the code for BlueGene architecture [J. Bigot] (Post-Doc MdS)
I Work in progress for WEST:→ Impact of W transport on turbulence [D. Esteve] (PhD)
→ Treatment of realistic geometry [A. Back] (Post-doc ANR GYPSI-CPT)
→ Aligned coordinates (kinetic electrons) [L. Mendoza] (PhD-IPP Garching)
, Virginie G N GENCI Gd Challenges N 21 May 2013 18
Collaborations:
I ANR GYPSI (2010-2014)→ Strasbourg, Nancy, Marseille
I ANR Nufuse G8@exascale (2012-2016)→ France, Germany, Japan, US, UK
I ADT INRIA Selalib (2011-2015)→ Strasbourg, Bordeaux
I IPL INRIA (march 2013-2017)→ Nice, Bordeaux
I New project following AEN INRIA Fusion(evaluation in progress)→ Strasbourg, Lyon, Nice
I Collaborations with IPP Garching(Germany) since 2012
I Collaborations with “Maison de laSimulation”- Saclay (Paris) since 2012
, Virginie G N GENCI Gd Challenges N 21 May 2013 19
top related