d. r. ernst (mit)...4 d. r. ernst monday, june 10, 2019 • hyperbolic equation that describes...

1D. R. Ernst Monday, June 10, 2019

SciDAC:David Hatch [Principal Investigator],M. Kotschenreuther, F. Jenko, C. Michoski (U. Texas-Austin);L. LoDestro, J. Parker (LLNL);W. Dorland (U. Maryland);G. Hammett, A. Hakim (PPPL); D. Ernst (MIT)

FASTMath:D. Reynolds (SMU)C. Woodward, D. Gardner (LLNL)M. Minion (LBNL)RAPIDS:S. Hongzhang, Lenny Oliker (LBNL)Courant:A. Cerfon (NYU)

mgkscidac.org

D. R. Ernst (MIT)FASTMath All-Hands Meeting, June 10-11 (2019), Argonne


Simulations of core turbulence in Tokamak Fusion Plasmas have matured:• Simulations have matched Particle, Momentum, Energy fluxes and Turbulence

Fluctuation Spectra simultaneously in experiments, within error bars [e.g., Ernst et al. Phys. Plasmas (2016)]

• Turbulent transport limits fusion power ∝ (pressure)2

Multiscale challenges:• Turbulence in the sharp

edge controls the core, but hard to simulate

• Electron thermal transport can involve turbulence scales spanning two orders of magnitude

• Evolving turbulence on 100x longer transport timescale to find temperature, density, rotation profiles

GENE gyrokinetic simulation of core turbulence


• Bridging the gap between turbulence and transport time scales for global turbulence [TANGO code]

• Ion-electron multiscale turbulence [GENE, Gkeyll codes]

• Practical algorithms for fast/efficient cross scale coupling [MuSHrooM code]

• Hermite-Laguerre gyrokineticcode [GX code]: seamless transition between fluid and kinetic to optimize rigor-efficiency

Exploit Multiscale Deal Directly with Multiscale

• Turbulence in transport barriers (H-mode pedestal, Internal transport barriers) [GENE, Gkeyll codes]

• Ion-electron multiscaleturbulence [GENE, Gkeyllcodes]

• Full multiscale simulations in H-mode pedestal

• New kinetic algorithms [Gkeyll]


• Hyperbolic equation that describes time-evolution of the density of particles f (x, y, z, v||, µ; t), in a 5D phase space – average over fast gyro-motion– Plus parabolic diffusion-drag terms in a collision operator term C[f] & elliptic field solves– 3 spatial dimensions (x, y, z); 2 velocity coordinates (v||, µ)– Typical moderate-big grids: 256 x 256 x 32 x 16 x 8 (~300M points, 1000’s of core-days)

• Describes turbulence like 3D Navier-Stokes/MHD equations, but in a 5D phase space

• To illustrate, ignore gyro-radius (long wavelengths), take straight magnetic field, ignore magnetic fluctuations (pressure≪ 8𝝅B2), use time-independent dielectric coefficient ϵ⊥0 (x):

• Can write the LHS of GK equation in Hamiltonian form

@f

@t+

@

@z

v||f

+r · (~vEf) +

@

@v||

q

mE||f

= C[f ] + S

r? · (?0r?) = 4gc = 4X

s

q

Zd3v f

= 4X

s

qB

Zdv||

Zdµfs(~x, v||, µ, t)

<latexit sha1_base64="7b413ipwT9dr00A9BlVXv5Q0QF0=">AAACn3icbVBNbxMxEHWWr1K+UjhyMURIRVpF2VAKl0IFB+CCgkTaomy0cryziVXb69qzaaPt/iZ+DQcu8Bf4CXg3qURaRrL05r0Zz8ybGCkc9no/W8G16zdu3tq4vXnn7r37D9pbDw9cXlgOQ57L3B5NmAMpNAxRoIQjY4GpiYTDyfH7Wj+cg3Ui119xYWCs2FSLTHCGnkran/boDo2NoLErVOLoCX1HY6GRpnSelOfn1UUWq4Jmidum8Rx4eVaFKz2slZDi86Td6XV7TdCrIFqBzts/pIlBstV6E6c5LxRo5JI5N4r6Bsclsyi4hGozLhwYxo/ZFEYeaqbAjcvm5oo+80xKs9z65xds2H87SqacW6iJr1QMZ+6yVpP/00YFZq/HpdCmQNB8OSgrJMWc1gbSVFjgKBceMG6F35XyGbOMo7d5bQpnmoNcu6OcQq4A7cKzxufN2uXpTGB974VYSkD/m2EGbCiZTh33MEQ4w1OR4mxvt/tS6Mo7Hl329yo46HejF93+l53Ofn9pPdkgj8lTsk0i8orsk49kQIaEk+/kB/lFfgdPgg/B52CwLA1aq55HZC2Cb38ByZLQuQ==</latexit>

~vE =1

B2~E ~B

<latexit sha1_base64="Spi7o2AMDrAFbr8mSYqeX17+SbA=">AAAChnicbVDbbhMxEHW2QEO5pfDIi0WExEO12t3Sy0tpVFSJxyKRtlISIq93NrHq9Vr2bNrI2t/ga3iFf+Av+AS8myKRlpEsnzlnxuM5qZbCYhT96gQbDx4+2uw+3nry9NnzF73tl+e2rAyHIS9laS5TZkEKBUMUKOFSG2BFKuEivfrY6BcLMFaU6gsuNUwKNlMiF5yhp6a9aLwA7hb19JQe0XFuGHdx7U6+JjVtlVN/oyjArtKTetrrR2HUBr0P4lvQP/5N2jibbnc+jLOSVwUo5JJZO4oTjRPHDAouod4aVxY041dsBiMPFfPTJq5draZvPZPRvDT+KKQt+2+HY4W1yyL1lQXDub2rNeT/tFGF+eHECaUrBMVXg/JKUixp4xPNhAGOcukB40b4v1I+Z94f9G6uTeFMcZBre7gZlAWgWXpW+7z9trueC2z2/Ss6Cehf00yD2ZFMZZZ7uINwg9ciw/nRfrgnVON4fNff++A8CePdMPn8vj9IVtaTLnlN3pB3JCYHZEA+kTMyJJx8I9/JD/Iz6AZhsBccrEqDzm3PK7IWweAP6gTI2Q==</latexit>

~E = r<latexit sha1_base64="PNRFzXuxCfCE30Xrj0sldOZBvpw=">AAACbXicbVBbaxNBFD5Zb7XeUkV8UGQwiD60IRuvL9WCCD5WMG0hG8LZ2ZPs0NnZYeZsa1j21/iqP8hfoT/ByaaCaT0w8M33neuXWq08DwY/O9Gly1euXtu4vnnj5q3bd7pbdw98WTlJI1nq0h2l6EkrQyNWrOnIOsIi1XSYHn9Y6ocn5LwqzRdeWJoUODdqpiRyoKbdB8kJyfpjI3bFjkgMphpFYnM17fYG/UEb4iKIz0Dv/W9oY3+61XmXZKWsCjIsNXo/joeWJzU6VlJTs5lUnizKY5zTOECDBflJ3V7QiKeBycSsdOEZFi37b0WNhfeLIg2ZBXLuz2tL8n/auOLZ20mtjK2YjFwNmlVacCmWdohMOZKsFwGgdCrsKmSODiUH09amSDSS9Nod9ZzKgtgtAmvDv127Ps0VL+/9K9aaOHSzaMltazSZlwFuM33lU5Vxvvu6/0qZJjgen/f3IjgY9uMX/eHnl7294cp62ICH8ASeQwxvYA8+wT6MQEID3+A7/Oj8iu5Hj6LHq9Soc1ZzD9YievYH5W2++Q==</latexit>

Poisson Equation, charge q

G. Hammett


• Core Eulerian Gyrokinetic Codes: small 1% fluctuations f1 about a Maxwellianequilibrium distribution f0 : f = f0 + f1

• Spatial gradients in equilibrium distribution drive instabilities

• Turbulence is quasi-2D in (x, y) and extended along the magnetic field

• Nonlinear term can be written as Poissan Bracket

• Typically use explicit or implicit time advance of f (kx, ky, z, v||, µ; t)

• Pseudo-Spectral nonlinear term evaluated in real space (2/3 de-aliasing rule)

• Lots of Bessel Functions J0, J1 for typical wavelengths＜gyro-radius

Life at the edge is tougher:• Very steep gradients, atomic physics, f1 / f0 ~ 30%: Global “full-f ” for ion scales• GKEYLL code is full-f, uses Discontinuous Galerkin methods

@f

@t+

@

@z

v||f

+r · (~vEf) +

@

@v||

q

mE||f

= C[f ] + S

, = @x

@y

@y

@x

<latexit sha1_base64="pvBlg/YyrodYwlQkSCQx4DaszK4=">AAAE0XiclVNNbxMxEHXbAGX5aAtHLharSgilbTY9wAWpEheORZC2UjaKvN7ZxIrtNbY3JFqthLjyw/gN/AiucMXerNRkCwIsWR6/N88ztmcSxZmxvd63re2dzq3bd3bvBvfuP3i4t3/w6MLkhaYwoDnP9VVCDHAmYWCZ5XClNBCRcLhMZq89fzkHbVgu39ulgpEgE8kyRol10HifxAl8kEFcxmrKujhWhsUVfuUMoi0jfLzAnom718iy9sJHa0jbZVG7BDG4s8f7Ye+4Vw9804gaI0TNOB8f7FzFTKrClicD41I/SUFLY08sLPysKQdq8NMSPhvTXAgiU1MF/yi0UzDM/L+OKNOL1mVpTgsB0lJOjBlGPWVHpX8DysGxhQFF6IxMYOhMSQSYUVn/WYUPHZLiLNduSotrdF1REmHMUiTOUxA7NW3Og7/l/naHKfA5uAxb+dns5ais3UDSVXpZwbHNsS8bnDIN1PKlMwjVzOkxnRJNqHXBNuIbK4he6rQKDtdhn6iywqPebGqyTHLe9rxOcB0eZjwn1nS9uGuJq2/Tzdik0G51Fa80k/7tlTnibAatI5VxrhsXbr4hiCV8bP7zeVlntiJK/7mQXSNhFK7xpFiUYVT9Ue/4ang6Kg8D7FpB59Y9XsyBzGEu8hQ8msCEyYnOC+V2bt9EKcN+FZ7WOpDpit8IU8ZGAa3cwiaCtKgEiGOGLRRqdFQFrhejdufdNC76x9Hpcf9tPzzrN125i56gp+gZitALdIbeoHM0QBR9Rd/RD/Sz866z7HzqfF65bm81msdoY3S+/AIaBqiS</latexit>

~vE =z r

B<latexit sha1_base64="+acbJTZ9CCgwnah2/oIyzBUojkM=">AAAEqHiclVNNbxMxEHXbAGX5auHIZcWqEqpKm00P9IJUgZA4Fom0Rdko8noniRWvbWxvaLD2r/BruMKdf8PsZqUmWxBgyfL4vXmesWecasGt63Z/bmxudW7dvrN9N7h3/8HDRzu7j8+tKgyDPlNCmcuUWhBcQt9xJ+BSG6B5KuAinb2p+Is5GMuV/OAWGoY5nUg+5ow6hEY7J0kKn2SQzIH5eTl6G74KfTKlzn8pE8dzsImkqaCJnvJE4UHh6zJIACWjnah72K1HeNOIGyMizTgb7W5dJlzqwvmjvsWMjjIw0rojB1fVrCkEDVTTUTEbMZXnVGYWI/6b0E3Bcvv/OqptN16VZYoVOUjHBLV2EHe1G3pqHGcCkC0saMpmdAIDNCXFVxr6uhRluIdIFo6VwSldWKOrCk9zaxd5ip45dVPb5irwt9zf7jAFMQfMsJWfG58Mfe0Gki3TGxcidCqsuiHMuAHmxAINygxHfcim1FDmMNhafOtyahYmK4O9VbhKVLu8QiuzaTWfKtH2vE5wFR6MhaLOHlTiA4etBvZgzCeFwRUbWRsuq7fX9oXgM2gdqS26rl24KUOQSPjc1HPf15ktCV8VF8bXSBRHKzwtrnwUl3/UI18Ojod+LwjDRBvl8PESAXQO81xlUKEpTLicGFVo3OG+ieKjXhkd1zqQ2ZJfC+MTq4GVuPBJTltUChSZQQuFGh2WAf7FuP3zbhrnvcP4+LD3vhed9ppfuU2ekmfkOYnJS3JK3pEz0ieMfCXfyHfyo7PfOetcdD4uXTc3Gs0TsjY66S+nH5ks</latexit>

r · (~vEf) = B1, f1+ ~vE ·rf0 + ...<latexit sha1_base64="pUdQZnxBZ2l8bCwd4ZoEq25IT7A=">AAAFB3iclVPdbtMwFE63AqP8bINLbiyiSQO6LunE3wXSBELickh0m9SUynFOWquOY2ynrLLyADwBj8Ed4pbH4GGQsNNMazsQYMnKyfed75xjH59YMKp0EPxorK03r1y9tnG9dePmrdubW9t3jlVeSAI9krNcnsZYAaMceppqBqdCAs5iBifx5JXjT6YgFc35Oz0TMMjwiNOUEqwtNNz6HMXwgbcijmOGI5LkejeaAjHTcvg6fYBeoJfvzV5YRhnW4yqbkZCUJjKRGNM2SodhVJboETJRxfblKB6YoBNUqx10Hgfh8ydh+xyxyvPoVbJ5XhsmqKJ0Op1WBK6g4ZZ/rkGXjbA2fK9eR8Pt9dOIclFos99T9sD7CUiu9L6GM7cryoIS3NaYTYYkzzLME1W2/lGox6Co+n8dFioIF2VJTooMuCYMK9UPA6EHBktNCQPLFgoEJhM8gr41Oc5ADUx1uyXasUiC0lzazTWq0EWFwZlSsyy2nq5japVz4G+5v51hDGwKtsKV+nT6bGAqN+BkXl5aMKRz5B4bSqgEotnMGphIavWIjLHERNtkS/mVzrCcyaRs7SzCrlChM4c6s37JJs7ZqudFgYtwP2U51qrtxG1t3xqodkpHhbRfOydCUu7uXqg9RiewElIo67p04LoNdlzgY93Ph+ZiNkrjmgvpBeKH/gKPizPjh+Uf9ZYv+wcDs9NCKBIy1/byIgZ4CtMsT8ChMYwoH8m8EPbP/tdZjN8t/YNKBzyZ80tpTKQEEDt+io4yvELFgC3TX0GhQgelm8VwdfIuG8fdTnjQ6b7t+ofdeio3vHvefW/XC72n3qH3xjvyeh7xfjb8Rrux1/zU/NL82vw2d11r1Jq73tJqfv8Fe9C2qA==</latexit>

@f0/@x<latexit sha1_base64="8bgu8wD2AFcOcg9YxPYX0yzmM7I=">AAAEuHiclVNdb9MwFPW2AiN8rINHXiKySQiVNunE59MkXvY4JLpNaqPiODepVcexbKessvKX+DW88AC/BSfNtjaAAEuWr8+5x/favjcSjCrt+9+2tnc6t27f2b3r3Lv/4OFed//RmcoLSWBEcpbLiwgrYJTDSFPN4EJIwFnE4Dyav6/48wVIRXP+US8FhBlOOU0owdpC0+7JgWMm9TFjmUah8ft+PXp+/6UfvH0V9K6Q0kwElppi5iZT3x2419vLsnQOpl3vytO90rrXSNAYHmrG6XR/52JCuSi0GYyUzXAQg+RKDzRcVrOmLCihmhqz+ZTkWYZ5rErnH4V6Boqq/9dhofxgXRbnpMiAa8KwUuPAFzo01eUJA8sWCgQmc5zC2JocZ6BCU79p6R5aJHaTXNrJtVuj6wqDM6WWWWQ9M6xnqs1V4G+5v91hBmwBNsNWfjp5E5raDThZpZcUzNW5W1WHG1MJRLOlNTCR1OpdMsMSE22DbcRXOsNyKePSOVyHq0SFziq0MpvSM1HO2p43Ca7D44TlWKteJe5pbMtY9RKaFtKutrCFpLx6e6FeMDqH1pFCWdeNCzff4Ew4fG7+87mpM1sRpvpcSG4QL/DWeFxcGi8o/6i3fDk+Cs2h49p2kLm2jzdhgBewyPIYKjSClPJU5oWwO7tvohhvWHpHtQ54vOI3wpiJEkBs0ymaZrhFRYAtM26hUKNh6dheDNqd96txNuwHR/3hh6F3PGy6chc9QU/RMxSg1+gYnaBTNEIEfUFf0Xf0o/Ou86mTdujKdXur0TxGG6MjfwJjGJn0</latexit>

~B = Bz<latexit sha1_base64="wTWvM9SU4/qXEeJ3xdXc4KoHkj4=">AAAEmXiclVPNbhMxEHbbACX8tXDsxWJbCVBJs8kBLkilXCpORZC2UjaKvN7ZxIrXa9ne0GDtM/A0XOE5eBvszUpNtiDAkuXx983nGY/HseRMm27358bmVuvW7Tvbd9v37j94+Ghn9/G5zgtFYUBznqvLmGjgTMDAMMPhUiogWczhIp698/zFHJRmufhkFhJGGZkIljJKjIPGO8/3cZQRM61OsjEvoMQ2mgO1JyV+g09wNCXGfinL/fFO0O10q4FvGmFtBKgeZ+PdrcuICVkYezTQLoWjBJTQ5sjAlZ8V5UAFfhrCZ2OaZxkRiS7b/yg0U9BM/7+OSN0NV2VJTosMhKGcaD0Mu9KMLFGGUQ6OLTRIQmdkAkNnCpKBHtmqYiU+cEiC01y5KQyu0FWFJZnWiyx2nr7Ousl58Lfc3+4wBT4Hl2EjP5O+HtnKDQRdppcWHJsc++fHCVNADV84g1DFnB7TKVGEGhdsLb42GVELlZTtg1XYJypN5lFv1r1l45w3Pa8TXIWHKc+J0YdefGiI61N9mLJJodzqOlcqJnztpX7J2QwaR0rtXNcuXD9DOxLwuX7PF/a6o0vrHxfSayQIgxWeFFc2CMs/6h1fDvsje9DGOJIqN654EQcyh3mWJ+DRGCZMTFReSLdz+zqKDXpl0K90IJIlvxbGRloCLd3CJhlpUDEQxwwbKFToqGy7vxg2f95N47zXCfud3odecNyrf+U22kNP0TMUolfoGJ2iMzRAFH1F39B39KO113rbOm29X7pubtSaJ2httD7+AiXbkmM=</latexit>


• Multiscale phenomena present opportunity and challenge– Opportunity: exploit scale separation to simplify– Challenge: need to address multiple scales directly

• Kinetic turbulence: evolve f(x,v,t) – Time scales: Turbulence vs Transport– Spatial scales:

• Equilbrium vs fluctuations• Ion vs Electron

– Phase space (scales in velocity space)– Low vs high order moments (i.e. fluid

vs kinetic)– MGK addresses multiscale issues in

all 5 dimensions


N. Howard, GYRO

Electron scale box

Slow propagation

• Exploit factor ~60 scale separation in y-direction and time to significantly reduce computation time for turbulent transport with cross-scale coupling [Ernst, Francisquez, Pan, Hammett, Hatch, Dorland, Jenko ]

• Non-local coupling between scales important in some scenarios

• Algorithm: Save time by not using fine electron y-grid & timestep for ion scales

IonScales

IonScales

ElectronScales

. . .Freeze Beat

WavesFreeze

Time • Use reduced model as test bed for

time-stepping algorithms such as Additive Runge-Kutta [Reynolds]

[Maeyama, Howard et al. heroic ~30M hour gyrokinetic simulations]

Ion scales shear Electron scales

Electron scales damp ion scale zonal flows and beat into ion scales, increasing transport


• Nonlinear physics can be modeled with Modified Hasegawa-Mima Equation

Growth inducing drive term (input)

@

@t+ ', + @'

@y= 0

<latexit sha1_base64="S9e/1nm+JSMzfce/Z0G39Sp1qR8=">AAAEvHiclVNdb9MwFPW2AiN8bfDIi0U0CUHZkk4CXkCTeOGBhyHRMampKse5ba3ajrGdsMrKn+LXwCP8Epw0aG0GAixZvj7nHt9r+95UcWZsFH3b2t7pXbt+Y/dmcOv2nbv39vbvn5m80BSGNOe5Pk+JAc4kDC2zHM6VBiJSDh/TxZua/1iCNiyXH+xSwViQmWRTRon10GTvXZLCJxkkKgPNSpcowypnK/wUJw4nJdFqzvq4hnHSoL8cV1TllhV+haMgAX/MZC+MDqNm4KtG3BohasfpZH/nPGFSFdYdDY3P8sgfLo09snBRz4byoIZ6WsIXE5oLQWRmquAfhXYOhpn/1xFlonhdluW0ECAt5cSYURwpO3ZEW0Y5eLYwoAhdkBmMvCmJADN2zfdU+MAjGZ7m2k9pcYOuKxwRxixF6j0FsXPT5Wrwt9zf7jAHXoLPsJOfnb4cu8YNJF2lNy04tjmuKwRnTAO1fOkNQjXzekznRBNqfbCN+MYKopc6q4KDdbhOVFlRo7XZlp9Lc971vExwHR5NeU6s6dfiviW+lE1/ymaF9qsvbqWZrN9emWecLaBzpDLedePC7TcEiYTP7X8+cU1mK8LVnwvTSySMwzWeFBcujKs/6j1fjY7H7iDAvjt0bv3jJRxICaXIM6jRFGZMznReKL/z+zaKCwdVeNzoQGYrfiOMS4wCWvmFzQTpUCkQz4w6KDTouAp8L8bdzrtqnA0O4+PDwftBePK87cpd9BA9Qo9RjF6gE/QWnaIhougL+oq+ox+9172st+iJlev2Vqt5gDZGr/wJSOWgUQ==</latexit>

= ' h'iy 0(k)@y'r2?'

<latexit sha1_base64="b0YpGQi1YDJVkkRuVYw9DWcrDRE=">AAAE4HiclVPPb9MwFE63bozwa4MjF4to0kDb2nQScEGaxIXjkOg2qS6R47ykVh3H2E5ZFeXODXHlD0Pif+GAnUas7UCAJcvP3/c+v2c/v1hypk2//72zsdnd2r61c9u/c/fe/Qe7ew/PdVEqCkNa8EJdxkQDZwKGhhkOl1IByWMOF/H0teMvZqA0K8Q7M5cwzkkmWMooMRaKdiWO4YPwsdQMvfLxjCg5YegIYU5ExgG1CFbNNpr7RzgBbkjURwfTp1gSZRjh0Rz9kmJBYk6iCktQsn4/aAkfgwvkR7tB/7jfDHTTCFsj8NpxFu1tXmImZGmq3lDbm/QSUEKbnoErNxvKggrcNIRPI1rkORGJrv1/FJoJaKb/X0ek7ofLsqSgZQ7CUE60HoV9acaVeyHKwbKlBknolGQwsqYgOehx1ZSwRvsWSVBaKDuFQQ26rKhIrvU8j61nTsxEr3MO/C33tztMgM/AZriWn0lfjqvGDQRdpJeWHJkCuV+EEqaAGj63BqGKWT2iE6IINTbYSnxtcqLmKqn9/WXYJSpN7lBntl+0igu+7nmd4DI8SnlBjD504kNjfxzow5RlpbKrbQCpmHBvL/URZ1NYO1Jq67py4bYMPhbwsa3ns6rJbEFUrriQXiNBGCzxpLyqgrD+o97y9ehkXO37CGGpCmMfD3MgM5jlRQIOjSFjIlNFKe3O7tsoVTCog5NGByJZ8CthKqwl0NouLMvJGhUDscxoDYUGHdeuF8P1zrtpnA+Ow5PjwdtBcPq87cod77H3xDvwQu+Fd+q98c68oUe9b96PzlZnuxt3P3U/d78sXDc6reaRtzK6X38CiRur9Q==</latexit>

, = @x

@y

@y

@x

<latexit sha1_base64="SFWp7Sgv8lbkMPjtzHvrB7mkVUA=">AAAE0niclVNNbxMxEHVpgLJ8tXDksmJVCaG0zaYScEGqxIVjkZq2UjaKvN7ZxIrtNbY3JFh7QFz5YfwF/gRXOGJvVmqyBQGWLI/fm+cZ2zOpZFSbXu/b1o3tzs1bt3fuBHfv3X/wcHfv0bkuSkVgQApWqMsUa2BUwMBQw+BSKsA8ZXCRzt54/mIOStNCnJmlhBHHE0FzSrBx0Hg3TVJ4L4LEJnJKu2EiNU2q8LUzsDIUs/Ei9EzSvUKWtVd4sIa0XRa1S5CAP3u8G/UOe/UIrxtxY0SoGafjve3LhApZGns00C73owyU0ObIwMLPmnKgAj8NZrMxKTjHItNV8I9CMwVN9f/rsNS9eF2WFaTkIAxhWOth3JNmZP0jEAaOLTVITGZ4AkNnCsxBj2z9aVW475AszAvlpjBhja4rLOZaL3nqPDk2U93mPPhb7m93mAKbg8uwlZ/JX41s7QaCrNLLSxaaIvR1E2ZUATFs6QxMFHX6kEyxwsS4YBvxteFYLVVWBfvrsE9UGu5RbzZFadOCtT2vElyHhzkrsNFdL+4a7Apcd3M6KZVbXclLRYV/e6kPGJ1B60ipnevGhZtvCBIBH5r/fG7rzFaE9Z8L+RUSxdEaj8uFjeLqj3rHV8Pjkd0PQtcLqjDu8RIGeA5zXmTg0RQmVExUUUq3c/smio36VXRc60BkK34jjE20BFK5hU44blEpYMcMWyjU6KjyvRi3O++6cd4/jI8P++/60cmLpit30BP0FD1DMXqJTtBbdIoGiKCv6Dv6gX52zjofO586n1euN7YazWO0MTpffgGO9Kiq</latexit>

• Preliminary model generalization to multiscale (2 orders of magnitude in k)

k = [0(k2) + 0(k

2 me/mi) 2 + k22De/2s iky0/s ] 'k 0(k

2)'ky=0<latexit sha1_base64="Ihw7Cw/vOOvPAb9Z/u0tU64Qx0E=">AAAFXniclVNtb9MwEE63FUZhbAMhIfHFIpq0QbemnQRI06RJDMHHIbEXqe4iJ7m2Vm0n2E5ZFeV38DP4PXzjp3BJM9Z2IMCSlfPz3JM7++6CRHBjPe97bWl5pX7n7uq9xv0Haw/XNzYfnZk41SGchrGI9UXADAiu4NRyK+Ai0cBkIOA8GL0t+PMxaMNj9clOEuhJNlC8z0NmEfI3vtEAPqsGTQz3Mzoe5eSQdGmTvmdSMt/bHl12dshLMnumTSJ9aEmf75Bd0iFIF6jAoBHzs2PILzstqoexby476MFH/oRGICzqK5w2e/SAjplOhlXYXTIfElO4plF+6OUNCkWi/obr7XnlIreNdmW4TrVO/M3lr5SrJLVZ69TgQ7Qi0MrYloWrYpcUghqKbZkY+WGMeajIYMh/E9ohGG7+X8cS47VnZVEcphKUDQUzptv2EtvLmLY8FIBsaiBh4YgNoIumYhJMLys7ICdbiESkH2vcypISnVVkTBozkQF6SmaHZpErwN9yf7vDEMQYMMOF/Gz/TS8r3UCF0/T6qSA2JkUTkohrCK2YoMFCzVFPwiHTLLQYbC6+sZLpiY7yxtYsXCSaWFmghVl1eBbEYtHzJsFZuNsXMbOmWYibluG0mGafD1KNX5yfRHNVvH1idgUfwcIvE4OucxeuytCgCr5U9XyRlZlNiawoLvRvELftzvAsvcrcdv5HPfJ5d7+XbTUIoYmOLT4eFcDGMJZxBAUawICrgY7TBE94rqJkbid390sdqGjKF7e5iZNRk0CY44cPJFvkAmBIdRdhKOHer06XWMXYYJ3JABQ0SZBaEgDhiog4ZKKo97XrUPqVN45ye3Fwbxtnnb32/l7nY8c9elUN9arzzHnubDtt57Vz5HxwTpxTJ6w9qR3UjmvvVn7U79TX6utT16VapXnszK3605+LdtLP</latexit>

0(k2) = I0(k

2) ek2

<latexit sha1_base64="6CSzvPAsAYwqSaEkpZbw1C50XyA=">AAAE33iclVPNb9MwFPdYgVG+NjhysYgmDdRtSScBF6RJHIDbkOg2qekqx3lNrfoj2E5ZFeXMDXEC8b9whf+B/wY7DazrQIAlKy+/j7znvOck58zYMPy+cmm1dfnK1bVr7es3bt66vb5x59CoQlPoUcWVPk6IAc4k9CyzHI5zDUQkHI6SyTPPH01BG6bkazvLYSBIJtmIUWIdNFwP4wTeyHb8nAhBhuHW5KT7AD/FL5sw7sRwUm67sGrH4JXD9SDcCeuFLwZREwSoWQfDjdVPMZN5YcvdnnGV7KagpbG7Fk79rikHavDbEj4ZUuVqkalxKf/NaMdgmPl/H8lNGC3aUkULAdJSTozpR2FuByXRllEOji0M5IROSAZ9F0oiwAzKugUV3nRIikdKuy0trtFFR0mEMTOROKUgdmyWOQ/+lvvbGcbAp+AqXKrPjp4MyloGks7LGxUcW4X9FOCUaaCWz1xAqGbOj+mYaEKtS3Yuv7GC6JlOq/bmIuwLza3wqA+bESsTxZeVZwUuwv0RV8Sajjd3LHHjajojlhXaPd0A55pJ/+9zs83ZBJY+mRsnPXfgpg3tWMLbpp8Py7qyOVH65sLoDAmiYIEnxWkZRNUf/Y6v+nuDcrONcZxrZd3PizmQKUyFSsGjCWRMZloVuXtz702WMuhWwV7tA5nOeX+aszxlbHKglXuwTJBlLgHiqP4yDDU8+DXpwnVRGddnnIGEDk4KixPATGKuKOG+3z+lYzFs1O4qR8sX92Jw2N2J9na6r7rB/qPmUq+he+g+2kIReoz20Qt0gHqIoo/oC/qKvrVI613rfevDXHpppfHcRedW6/MP6GmqjQ==</latexit>

D. R. Ernst

• Nonlinear term is Poisson Bracket of same form as gyrokineticequation

• Expedient test-bed for multi-rate and multi-scale algorithms• Transfer successful methods to gyrokinetic codes (GENE, GKEYLL)

• Bessel functions arise due finite gyroradius


M. Francisquez, Jay Lang, D. R. Ernst, G. W. Hammett

• Standard 2D pseudo-spectral solver for φ( t, kx, ky ) using 2/3 de-aliasing rule, FFTW, and various time advance schemes, various hyperdiffusivities

• Use as test-bed for time-stepping algorithms

• Test ARKode for sub-cycling electron scales in collab. with D. Reynolds

• Expect dramatic speedups via higher order accuracy from ARK, allowing larger timesteps

• Opportunities for exploiting spatial scale separation in y

(Without ZF term)


• Same parameters and resolution used in [Smith & Hammett, Phys. Plasmas (1998)] – Resolution is 128x128 in aliased real space, or 43x43 in k-space not counting negative kx's

• In both the spectrum and the growth rates the high-k trends are a little different. – We do not have the same hyperdiffusion model.– Different time-stepping algorithm, which introduces a different form of diffusion.– Method for computing averages over k-space might differ.

Particle Flux Spectral Intensity

M. Francisquez, Jay Lang, D. R. Ernst, G. W. Hammett


• Continuum approach to gyrokinetics(evolve distribution function on grid).

• Publicly available, world-wide user base from ~30 scientific institutions (US), ~100 worldwide

• Modes of operation:– delta-f & full-f (gradient-driven, flux-driven)– flux-tube & full-flux-surface & global

• Unique combination of various FDM and spectral methods

• Extensive physics: kinetic electrons, electromagnetic effects, collisions, realistic MHD equilibria, electron-scale turbulence…

• Part of fusion whole device modeling ECP project

GENE on top-level HPC resources

INCITE Award (2016)

Speedup with respect to 2048 nodes

Number of Titan Nodes (16 CPU cores + 1 GPU)

Spee

dup

163844096 819220481

2

4

8

Strong scaling of GENE on Titan (2k-16k nodes)


• Working with Shan Hongzhang(LBNL) targeting Cori KNL

• GENE already scales well on Cori but improving on-node optimization

• Improved communication performance 15% by avoiding user defined MPI data types for non-contiguous data

• Extensive studies of FFT performance

• Work going forward will focus on GPU optimization– RAPIDS work targeting NERSC 9 – ECP targeting SUMMIT

• Strong scaling of GENE on Cori

Time

per s

tep

(s)

# MPI Processes

S. Hongzhang, F. Jenko, D. Hatch et al.


• Conservative and symmetric structure enables a finite-volume or spectral discretization that preserves the conservation laws

gyro-tensors for test-particle (field-particle) drag and diffusion coefficients(can be pre-computed;; constant in time)

Q. Pan and D. R. Ernst, Phys. Rev. E 99 (2019)


• Initially implemented1 in the GENE gyrokinetic code using same finite-volume method2 as the latest model operator (Sugama operator)

• Conservation to machine precision independent of resolution; H-theorem• Zonal flow damping ~40% faster than Sugama model at high collisionality

• Initial nonlinear runs completed comparing exact/model

• Opportunity to speed up with FASTMath & GPUs

Convergence with FLR effects

requires >100 µ’s(8-16 without)

1Q. Pan and D. R. Ernst, to be submitted2P. Crandall et al., submitted to JCP (2018)


• Spectral methods for 2D velocity space; significantly less grid points– Formulate collision operator in weak (Galerkin) form

– Expand distributions and test functions in polynomial bases

– Use quadrature rule to perform velocity integration

– are constant and precomputable; requires operations

• Multi-rate time integrator for collisionless dynamics and collisions (tokamak core parameters); IMEX; Fast Multipole or Krylov-Implicit methods for collisions [D. Reynolds, C. Woodward]

• Nonlinear drift-kinetic implementation in GKEYLL edge code.


• The Problem: solve for temperature profile 𝑇 𝑟 in a tokamak, balancing between input heating and turbulent heat losses– Turbulent flux computed by GENE,

which depends on 𝑇 𝑟 , etc. (thousands of cores, tens of hours)

– Balance equation is solved with a code TANGO

ITERATION 4

SOLUTION

INITIAL

10

7

3

2

1

00.0 0.2 0.4 0.6 0.8

Minor Radius (r/a)Te

mpe

ratu

re (k

eV)

T(r/a)

HEAT SOURCE

Nonlinear Problem

Iterative Solution

Current solution method is a variant of fixed-point iteration

Jeff Parker (LLNL)Lynda LoDestro (LLNL)

Parker, LoDestro, Told, Merlo, Ricketson, Campos, Jenko, Hittinger, Nucl. Fusion, 58 054004 (2018).


• Every iteration requires a call to a massive turbulence simulation– 50-100 iterations in the course of a single GENE run are required currently

(see Parker et al.)– Can be expensive!

• Are there advanced methods, such as Anderson Acceleration, to reduce the number of iterations required?

• We’ve had discussions with Carol Woodward and David Gardner (LLNL) about using KINSOL’s robust Anderson Acceleration solver– Looks promising as an area to pursue

Jeff Parker (LLNL)Lynda LoDestro (LLNL)


• Burning plasmas will need an edge transport barrier – ITER, e.g., will need ~4 keV pedestal– Need to understand and predict– Recent breakthroughs in pedestal

turbulence with GENE [Hatch et al NF 2016, 2017, 2018, 2019; Kotschenreuther et al NF 2017, 2019]

• Goals– Develop reduced models / surrogates – UQ with reduced models– Ion-Electron multiscale treatments

DENSITY

ELECTRON TEMPERATURE

D. Hatch, M. Kotschenreuther (Univ. Texas)


• Map linear simulations (cheap) to nonlinear transport levels (train on high-fidelity simulation data)

• Further refine in comparison with experimental data

• Needs:

– Machine learning or advanced statistical techniques to define mapping

– Efficient algorithms for exploring a high-dimensional parameter space

– ML / statistical approaches to learning from both high-fidelity sims and experimental observations

Reliable reduced models for pedestal transport currently do not exist

D. Hatch, M. Kotschenreuther (Univ. Texas)

• Ion-electron multiscale: substantial transport at ion scales and electron scales.

• So far no tests of cross-scale coupling (likely very different than core multiscale).

Single scaleETG simulation

Single scaleIon scale

Neoclassical


Edge region very difficult computationally.

XGC only code crossing last closed flux surface at present.

E. Shi, A. Hakim, G. Hammett, et al. J. Plasma Physics (2017), T. Bernard et al. Phys. Plasmas (2019). Q. Pan et al. Phys. Plasmas (2018), similar work in straight fields with GENE.

Simplified helical model of SOL (toroidal + vertical B field) – general B-field geometry under test.

MGK SciDACG. W. Hammett, A. Hakim, M. Francisquez, et al.


• Novel version of Discontinuous Galerkin (DG) algorithm– Conserves energy for Hamiltonian system even with upwind fluxes

[Juno, Hakim, et al. JCP 2018] – High-order local algorithms reduce communication costs, helpful for Exascale.

• New modal version ~30x faster than nodal version– Computer-algebra generated code (w/ Maxima) makes use of sparseness of modal

interactions.

• Framework: LuaJIT over C++, uses ADIOS, Eigen, MPI, …

• 3 Main Versions, used in 3 SciDACs:1. Gyrokinetic DG version for edge turbulence in fusion,

in MGK SciDAC project (D. Hatch, PI) for pedestal / multiscale work,in HBPS SciDAC project (C.S. Chang, PI) for scrape-off-layer turbulence work.

2. Vlasov-Maxwell/Poisson DG version: solar wind turbulence (PU/Maryland), plasma-surface interactions in thrusters (AFOSR / Virginia Tech) & tokamak disruption SciDAC (LANL / Virginia Tech)

3. Multi-moment multi-fluid (extended MHD) finite-volume version: reconnection (Princeton Center for Heliophysics), global magnetosphere simulations (UNH)


• In HBPS SciDAC, collaborating w. ORNL applied math group (Cory Hauck & Eirik Endeve) on RK-Legendre super-time stepping for collision operator (diffusion-advection operator in velocity coordinates).

Many opportunites for more collaborations:

• Efficient implicit solver for parallel electron dynamics. – What would work? Most iterative implicit methods for parabolic/elliptic problems, but fast

electron motion is a hyperbolic operator: high-frequency wave needs to be implicit for stability, but not important itself.

• Improve scalability of IO—issues beyond a few thousand cores. Want to discuss with ADIOS group.

• Help with performance tuning, starting with hotspot analysis.

• Porting to GPU

• Post-processing, visualization, in-situ analysis

• Better parallel Poisson solvers when we get to larger systems (was using PETSc, now Eigen)


• GX code (Dorland et al)– Laguerre / Hermite in velocity space– Can use intelligent closures and be

gyrofluid when applicable– Alternatively can keep lots of moments

and be fully gyrokinetic– Status:

• Collision operator implemented• Linear closures tested • Extensive linear benchmarks• Nonlinear version runs on GPUs

• Developing ETG edge model• Spectral deferred correction in

collaboration with M. Minion (LBNL)

• Goals:– Seamless transition between fluid and

kinetic descriptions • Depending on physical regime,

desired speed / accuracy– Use machine learning to train fluid

closures from kinetic simulations– Start with toy problem then apply to GX

[Mandell, Dorland, Landreman, JPP 2018]

Wavenumber

Line

ar G

row

th R

ate

Temperature Gradient


• Cross-validation studies between– Simulation codes – Simulations and experiments– Different experiments

• Advanced data analytics– Real-time diagnostic tools– Machine learning algorithms– Data mining of simulations and

experiments

• Consistent and rigorous uncertainty quantification

• Multifidelity hierarchy design

• Model adequacy and prediction assessment capabilities

• Validation, optimization, design, data mining

MGKDB

MultifidelityModeling

Sensitivity Analysis

Model Adequacy

Cross-Validation

Advanced Analytics

Uncertainty Quantification

Craig Michoski (ICES, Univ. Texas)

Unified and cross-referenced database of experimental and model sensitivities


• Postdoc on the way

• Early stages. MongoDB prototype developed. Running locally and on NERSC.

• RAPIDS (i.e. Tahsin Kurc, Kshitij Mehta, …) interested in interfacing a DB through ADIOS for performance and large data transfer, and helping to facilitate this

• Existing codes already connected with MGKDB (e.g. GENE, GS2, GX, GYRO, QuallKiz, etc.)

• Existing Open Source European initiative from Jonathon Citrin, Karel van de Plassche, Yann Camenen, called GKDB: https://github.com/gkdb/gkdb.git– Focused on ITER Integrated Modelling & Analysis Suite (IMAS)– OMFIT integration– PostgreSQL relational database– Strong interest in combining forces to maximize utility/workforce for the community at large

Craig Michoski (ICES, Univ. Texas)


• [MuSHrooM, GENE] D. Reynolds (SMU), C. Woodward (LLNL) – Multi-rate methods (ARKode) for multiscale turbulence, collisions

• [TANGO] Carol Woodward and David Gardner (LLNL) – KINSOL’s Anderson Acceleration solver

• [GX] M. Minion (LBNL)– Spectral Deferred Correction in fluid-kinetic code

• [GENE] Shan Hongzhang (LBNL) – Optimizing Cori KNL

• [MGKDB] Tahsin Kurc, Kshitij Mehta, … (RAPIDS)– Interface DB through ADIOS for performance and large data transfer


• Spatial scale separation in multi-scale turbulence

• Implicit Krylov methods for collisions

• Machine learning or advanced statistical techniques to define mapping

• Efficient algorithms for exploring a high-dimensional parameter space

• Efficient implicit solver for parallel electron dynamics.

• Improve scalability of IO [ADIOS]

• Help with performance tuning, starting with hotspot analysis.

• Porting to GPU

• Post-processing, visualization, in-situ analysis

• Better parallel Poisson solvers

Thank You !

d. r. ernst (mit)...4 d. r. ernst monday, june 10, 2019 • hyperbolic equation that describes...

Documents