![Page 1: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/1.jpg)
Recent Developments in the NRL Spectral Element Atmospheric Model
(NSEAM)*
Francis X. Giraldo
www.nrlmry.navy.mil/~giraldo/projects/nseam.html
*Funded by the Office of Naval Research
![Page 2: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/2.jpg)
Motivation
Mission of NRL-MRY:
• Develop NWP models for the Navy
Computing Resources:
• High Performance Computing has shifted towards distributed-memory architectures comprised of tens of thousands of processors (e.g., clustered systems). Current fastest system has 32,000 processors (BlueGene by IBM).
Exploiting New Architectures:
• Need numerical discretization methods that decompose the domain naturally into many smaller subdomains
• Each subdomain can then be solved independently on-processor
![Page 3: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/3.jpg)
Motivation (continued)
State-of-the-Art Global Forecast Models:• Based on the spectral transformspectral transform method (i.e. spherical
harmonics)• Spectral transform method uses global approximation
functions and Nproc scales like O(T) and costs O(T^3)
New Model:• NSEAM is based on the spectral elementspectral element method which
combines the high-order accuracy of spectral transforms with domain decomposition property of finite elements
• Uses local approximation functions and Nproc scales like O(T^2) and costs O(N T^2)
![Page 4: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/4.jpg)
Domain Decomposition
ST Models: Nproc = O(T) NSEAM: Nproc=O(T^2)
![Page 5: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/5.jpg)
Spectral Transform Method(Legendre Modal Functions)
-1 -0.5 0 0.5 1-1
0
1
x
y(x
)
y1
y2
y3
y4
y5
![Page 6: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/6.jpg)
Spectral Element Method(Legendre Nodal Functions)
-1 -0.5 0 0.5 1-1
0
1
x
y(x
)
y1
y2
y3
y4
y5
![Page 7: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/7.jpg)
What We Need for a New Model
To construct a model the following things are required:
1. Governing set of Partial Differential Equations along with initial and boundary conditions (PDE)
2. A Coordinate System (CS)
3. A Grid Generator in order to place solution values within the domain (GG)
4. A Spatial Discretization method in order to approximate derivatives (SD)
5. A Time-Integrator in order to advance the solution in time (TI)
6. A Very Large Computer (VLC)
![Page 8: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/8.jpg)
Description of NSEAM
• Hydrostatic primitive equations (PDE)
• Equations formulated in Cartesian and sigma coordinates (CS)
• Spectral Element method in horizontal (SD)
• Finite Volume method in vertical (SD)
• 2nd order semi-implicit in time (TI)
• GMRES with block Jacobi Preconditioning (TI)
• Message-Passing Interface (VLC)
• Can use all types of grids including quadrilateral and triangular adaptive grids (GG)
![Page 9: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/9.jpg)
pcP
Hydrostatic Primitive Equations
0
u
t
)()(2
2uS
Pcuk
a
zuuu
t
up
)(
Sut
(Mass)
(Momentum)
(Energy)
(State)
)(qSq
qut
q
(Tracers such as
Moisture/Salinity)
![Page 10: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/10.jpg)
• Primitive Spherical Form:
• Pole problem can be avoided by:– Using Gauss quadrature (as in spherical harmonics)– Creating a hole in the domain (as in gridpoint models)– Using Cartesian coordinates (NSEAM)
0cos
cos
1
vu
at
Why Cartesian Coordinates?
![Page 11: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/11.jpg)
• Primitive Equations:
• Write Primitive Equations as:
• Weak Problem Statement: Find
– such that
– where
0)(
qSt
qqL
)(1 Hq
0 dqLy
e
N
e
e
1
y
SE Method
)(qSt
q
![Page 12: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/12.jpg)
• Polynomial space for quads and tris
• Polynomial Approximation on the Quadrilateral:
– With polynomial space
• Polynomial Approximation on the Triangle:
– With polynomial space
i
N
iii
N
ii qqzyxzyxq
P
2)1(
11
),(ˆ),,(),,( yy
SE Method: Basis Functions
i
NN
iii
N
ii qqzyxzyxq
P
)2)(1(
2
1
11
),(ˆ),,(),,( yy
},0|{ NjiP jiN
}0|{ NjiP jiN
NN PP
![Page 13: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/13.jpg)
Accuracy of SE Method (Shallow Water on Sphere)
![Page 14: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/14.jpg)
Icosahedral TelescopingHexahedral
Icosahedral Lat-Lon Adaptive
Grids
Lat-Lon
Thin/Reduced
![Page 15: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/15.jpg)
• For the system:
• There are 2 classes of possible time-integrators:– Eulerian - fixed reference frame– Lagrangian - moving reference frame
• There are 2 subclasses of these methods:– Explicit – strict time-step restriction– Semi-Implicit – lenient time-step restriction– Fully Implicit – no time-step restriction
)(qSt
q
Time-Integrators
![Page 16: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/16.jpg)
• Start with:
• Discretize by 2nd order TI
– Everything on the RHS is known. Very fast to solve because only requires the evaluation of a vector of length 3*Np*Nlev (detail of this approach can be found in MWR 2004 paper)
)(qSt
q
Explicit Time-Integrator
mn
mm
mn
mm
n qStqq
)(2
0
2
0
1
![Page 17: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/17.jpg)
• Start with:
• Discretize by 2nd order TI
– A global sparse NONLINEAR system must be solved. This can be quite expensive! The size of the system is (3*Np*Nlev) x (3*Np*Nlev)
)(qSt
q
Fully-Implicit Time-Integrator
mn
mm
nn qqtSq
2
0
11 )(
![Page 18: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/18.jpg)
• Start with:
• Rewrite as:
• Discretize by 2nd order TI
– A global sparse LINEAR system must be solved. Via a vertical mode decomposition we can do this by solving: Nlev systems of Np x Np sparse linear systems. This is very fast! (details of this approach can be found in QJRMS 2005 paper).
)(qSt
q
Semi-Implicit Time-Integrator
)()()( qLqLqSt
q
mn
mm
mn
mm
mn
mm
n qtLqStqq2
1
2
0
2
0
1 )(
![Page 19: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/19.jpg)
Performance for T239 L30 (DT=300 seconds on IBM SP4)
(Spectral Model)
(SE Model)
![Page 20: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/20.jpg)
Performance of NSEAM (216 Processors on IBM SP4)
T159 L24
T239 L30
T580 L60
![Page 21: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/21.jpg)
NOGAPS T80 L24
Rossby-Haurwitz Wave 4(T=5 days)
![Page 22: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/22.jpg)
NSEAM T80 L24
Rossby-Haurwitz Wave 4(T=5 days)
![Page 23: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/23.jpg)
ECMWF T106 L30
Held-Suarez Test Case(T=1200 days)
![Page 24: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/24.jpg)
NSEAM T64 L20
Held-Suarez Test Case(T=1200 days)
Zonal Wind (m/s) Temperature (K)
![Page 25: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/25.jpg)
Comparison of the explicit (dashed line) and semi-implicit (solid line) NSEAM. The semi-implicit uses a time-step 8 times larger than the explicit. The difference between the two are virtually indistinguishable which is what one would hope to see.
Jablonowski-Williamson Balanced Initial State(Surface Pressure Error for NSEAM T185 L26)
![Page 26: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/26.jpg)
-90 Meridian 0 Meridian
The terrain field for NSEAM T239 from two different views of the globe. In this test an initially-balanced still atmosphere is integrated for 30 days. The atmosphere remained perfectly balanced for the length of the integration which is exactly what should happen.
Initially-Balanced Still Atmosphere(NSEAM T239 L30 with terrain after 30 days)
![Page 27: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/27.jpg)
No Terrain Terrain
The viewpoint of these plots is from the –120 meridian. The temperature contours look similar for both but the effects of the Rockies, Andes, and Antarctica are clearly visible on the right panel.
Held-Suarez Test Case(Surface Temp for NSEAM T185 L24 after 30 days)
![Page 28: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/28.jpg)
Pressure Temperature
U Velocity V Velocity
Jablonowski-Williamson Baroclinic Instability(Surface Values at Day 9 for NSEAM T185 L26)
![Page 29: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/29.jpg)
Jablonowski-Williamson Baroclinic Instability
![Page 30: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/30.jpg)
Jablonowski-Williamson Baroclinic Instability(Surface Values for NSEAM T185 L26 during 0-30 days)
Pressure Temperature
![Page 31: Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo giraldo/projects/nseam.html *Funded](https://reader036.vdocument.in/reader036/viewer/2022062309/5697bf7d1a28abf838c84b7a/html5/thumbnails/31.jpg)
Future Work
• Further testing of NSEAM with terrain
• Physical parameterization (sub-grid scale processes)
• Non-hydrostatic equations (to better handle dx < 10 km)
• Non-hydrostatic Coastal Ocean Model (underway)