high-dimensional nonlinear data assimilation with the nonlinear
TRANSCRIPT
High-Dimensional Nonlinear Data Assimilation with the Nonlinear Ensemble Transform Filter (NETF)
and its Smoother Extension
Paul Kirchgessner, Lars Nerger Alfred Wegener Institute, Bremerhaven, Germany
Julian Tödter, Bodo Ahrens University of Frankfurt, Frankfurt, Germany
5. International Symposium on Data Assimilation, Reading, UK, July 18 – 22, 2016
Nonlinear Ensemble Transform Filter & Smoother
Overview
! Study new Nonlinear Ensemble Transform Filter – NETF (Tödter & Ahrens, MWR, 2015)
! Extend NETF for smoothing
! Test filter and smoother in realistic high-dimensional idealized ocean data assimilation experiments
Nonlinear Ensemble Transform Filter & Smoother
• represent state and its error by ensemble of m states
• Forecast: • Integrate ensemble with numerical model
• Analysis: • update ensemble mean
• update ensemble perturbations
(both can be combined in a single step)
• Ensemble Kalman filters & NETF: Different definitions of
• weight vector
• Transform matrix
Ensemble filters – ensemble Kalman filters & NETF
X
w
W
x
a = x
f +X
0fw
X0a = X0fW
Nonlinear Ensemble Transform Filter & Smoother
• Ensemble Kalman: • Transformation according to KF equations
• NETF (Tödter & Ahrens, MWR, 2015) ! Mean update from Particle Filter weights: for all particles i
Nonlinear ensemble transform filter - NETF
! Ensemble update • Transform ensemble to fulfill analysis covariance
(like KF, but not assuming Gaussianity) • Derivation gives
( : mean-preserving random matrix; useful for stability) (Almost same formulation: Xiong et al., Tellus, 2006) ⇤
W =pm
⇥diag(w)� wwT
⇤1/2⇤
wi ⇠ exp
⇣�0.5(y �Hxf
i )TR�1
(y �Hxfi )
⌘
Nonlinear Ensemble Transform Filter & Smoother
• Smoother: Update past ensemble with future observations • Rewrite ensemble update as
• Filter:
Ensemble Smoothers – ETKS & NETS
Xak|k = Xf
k|k�1Wk
analysis time Observations used up to time
• Smoother at time
! works likewise for ETKS and NETS ! also possible for localized filters
Xai|k = Xf
i|k�1Wk
i < k
See, e.g., Nerger, Schulte & Bunse-Gerstner, QJRMS 140 (2014) 2249–2259
Nonlinear Ensemble Transform Filter & Smoother
• Performance for small model (Lorenz-96) • In Tödter & Ahrens (MWR, 2015)
• NETF beats ETKF for m=20 and larger
Performance of NETF – Lorenz-96
How do NETF and NETS perform in a more realistic case?
update mechanism seems to be more beneficial than itsstochastic counterpart.The next experiment concerns the L2005 model,
which exhibits a distinct spatial structure compared tothe L96 model. Figure 5 shows the analysis error withrespect to ensemble size. Concerning the relative per-formances of the KF-based filters, the general structureis very similar to the L96 experiment. The most re-markable, seemingly counterintuitive difference is thathere the NLEAF1 performs better than the NLEAF2,except whenm5 100. Lei and Bickel (2011) do not showthe NLEAF2 for their larger-dimensional experiments,hence, we cannot directly compare these findings totheir results. A possible explanation could be revealedby the update formalism of theNLEAF2 [Lei andBickel2011, their Eq. (3)], which requires the estimation of anindividual analysis covariance matrix for each ensemblemember, based on each of the m perturbed obser-vations. These low-rank approximations of the d 3 dcovariance matrix with m members are subject to sam-pling error, and it seems that the stochastic errors of theperturbed observations amplify the errors in the
estimation of the covariances, which may lead to theseunexpected results in certain larger-dimensional cases.This hypothesis is supported by the fact that in the low-dimensional L63 experiments the NLEAF2 consistentlyoutperformed the NLEAF1. Additionally, in the L2005system, spatial correlations are more important than inthe L96 system, hence a reliable estimation of the co-variances is of greater relevance here. The NETF, whichalso focusses on the second-order statistics, does notsuffer from this issue. Again, it exhibits the smallestanalysis error form$ 20. We conclude that, particularlyin larger-dimensional scenarios, the benefits of the de-terministic update mechanism of the NETF becomemore apparent.To strengthen these findings, Table 3 gives an overview
of more diagnostic measures for m 2 f10, 25, 50, 100g.As in the L63 case, the comparison of RMSE and spreadas well as of innovation variance and expected innovationvariance indicate that, thanks to the inflation procedure,the filters behave reasonably well in both state and ob-servation space. Except for m5 10, all scores reveal thatthe NETF performs best. The CRIGN shows that in
FIG. 4. Result of the L96 experiment with double exponential observation errors withs2obs 5 1. Shown is the average RMSE for the six ensemble filters against ensemble size, i.e.,
NETF (black line), ETKF (black, dotted), ETKFrot (black, dashed), EnKF (gray line),NLEAF1 (gray, dotted), and NLEAF2 (gray, dashed).
FIG. 5. As in Fig. 4, but for the L2005 experiment with double exponential observation errorswith s2
obs 5 1.
1360 MONTHLY WEATHER REV IEW VOLUME 143
Nonlinear Ensemble Transform Filter & Smoother
Assimilation into NEMO
European ocean circulation model
Model configuration
• box-configuration SEABASS
• ¼o resolution
• 121x81 grid points, 11 layers (state vector ~300,000)
• wind-driven double gyre (a nonlinear jet and eddies)
• medium size SANGOMA benchmark
True sea surface height at 1st analysis time
Longitude (degree)
Latit
ide
(deg
ree)
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
True sea surface height at last analysis time
Longitude (degree)
Latit
ide
(deg
ree)
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
www.data-assimilation.net
Nonlinear Ensemble Transform Filter & Smoother
PDAF: A tool for data assimilation
PDAF - Parallel Data Assimilation Framework " a program library for data assimilation " provide support for ensemble forecasts " provide fully-implemented filter and smoother algorithms
(LETKF, LSEIK, LESTKF, …) " easily useable with (probably) any numerical model
(applied with NEMO, MITgcm, FESOM, MPI-ESM, HBM) " makes good use of supercomputers " first public release in 2004; continued development
Open source: Code and documentation available at
http://pdaf.awi.de
L. Nerger, W. Hiller, Computers & Geosciences 55 (2013) 110-118
Nonlinear Ensemble Transform Filter & Smoother
Online coupling: Minimal changes to NEMO
Add to mynode (lin_mpp.F90) just before init of myrank #ifdef key_USE_PDAF CALL init_parallel_pdaf(0, 1, mpi_comm_opa) #endif
Add to nemo_init (nemogcm.F90) at end of routine #ifdef key_USE_PDAF CALL init_pdaf() #endif
Add to stp (step.F90) at end of routine #ifdef key_USE_PDAF CALL assimilate_pdaf() #endif
Modify dyn_nxt (dynnxt.F90) #ifdef key_USE_PDAF IF((neuler==0 .AND. kt==nit000).OR.assimilate) #else
Aaaaaaaa
Aaaaaaaa
aaaaaaaaa
Start
Stop
Initialize Model generate mesh Initialize fields
Time stepper consider BC
Consider forcing
Post-processing
init_parallel_pdaf
Do i=1, nsteps
init_pdaf
assimilate_pdaf
Nonlinear Ensemble Transform Filter & Smoother
Observations and Assimilation Configuration
Observations
• Simulated satellite sea surface height SSH (Envisat & Jason-1 tracks), 5cm error
• Temperature profiles on 3ox3o grid, surface to 2000m, 0.3oC error
Data Assimilation
• Ensemble size 120
• ETKF and LETKF
• Localization: weights on matrix R-1 (Gaspari/Cohn’99 function, 2.5o radius)
• Assimilate each 48h over 360 days −60 −55 −50 −45 −40 −35 −30
2428
3236
4044
(a) SSH [m]
Longitude [°]
Latit
ude
[°]
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● −0.6
−0.4
−0.2
0.0
0.2
0.4
● ● ● ● ● ● ● ●
5000
4000
3000
2000
1000
0
(b) T [°C]
Latitude [°C]D
epth
[m]
24 26 29 32 35 38 41 44
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0
2
4
6
8
10
12
FIG. 3. Observation characteristics on day 8: (a) The horizontal domain is shown, together with the Argo
profiler locations (crosses) and the synthetic SSH observations (colored) on the Envisat tracks (thin lines). (b)
The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles along the
� = �50� longitude line. Note that at � = 44�, the true temperature field is zero due to the lateral boundary
conditions.
774
775
776
777
778
41
FIG. 2. Observation characteristics on day 8: (a) The horizontal domain is shown, together with the Argo
profiler locations (crosses) and the synthetic SSH observations (colored) on the Envisat tracks (thin lines). (b)
The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles (46 values
each) along the � = �50� longitude line. At � = 44�, the true temperature field is zero due to the lateral
boundary conditions.
817
818
819
820
821
42
−60 −55 −50 −45 −40 −35 −30
2428
3236
4044
(a) SSH [m]
Longitude [°]
Latit
ude
[°]
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● −0.6
−0.4
−0.2
0.0
0.2
0.4
● ● ● ● ● ● ● ●
5000
4000
3000
2000
1000
0
(b) T [°C]
Latitude [°C]
Dep
th [m
]
24 26 29 32 35 38 41 44
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0
2
4
6
8
10
12
FIG. 3. Observation characteristics on day 8: (a) The horizontal domain is shown, together with the Argo
profiler locations (crosses) and the synthetic SSH observations (colored) on the Envisat tracks (thin lines). (b)
The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles along the
� = �50� longitude line. Note that at � = 44�, the true temperature field is zero due to the lateral boundary
conditions.
774
775
776
777
778
41
FIG. 2. Observation characteristics on day 8: (a) The horizontal domain is shown, together with the Argo
profiler locations (crosses) and the synthetic SSH observations (colored) on the Envisat tracks (thin lines). (b)
The vertical grid of 11 layers is visualized, and embedded are the artificial Argo temperature profiles (46 values
each) along the � = �50� longitude line. At � = 44�, the true temperature field is zero due to the lateral
boundary conditions.
817
818
819
820
821
42
Nonlinear Ensemble Transform Filter & Smoother
Application of LETKF
True sea surface height at 1st analysis time
Longitude (degree)
Latit
ide
(deg
ree)
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
Initial guess of sea surface height
Longitude (degree)
Latit
ide
(deg
ree)
Estimated SSH at 1st analysis time
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
Nonlinear Ensemble Transform Filter & Smoother
Application of LETKF (2)
Estimated SSH at last analysis time
Longitude (degree)
Latit
ide
(deg
ree)
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
True sea surface height at last analysis time
Longitude (degree)
Latit
ide
(deg
ree)
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
Final SSH without assimilation True sea surface height at last analysis time
Longitude (degree)
Latit
ide
(deg
ree)
−60 −55 −50 −45 −40 −35 −30
24
28
32
36
40
44−0.6
−0.4
−0.2
0
0.2
0.4
0.6
Nonlinear Ensemble Transform Filter & Smoother
• RMS errors reduced to 10% (velocities to 20%) of initial error • Slower convergence for NETF, but to same error level as LETKF • CRPS (Continuous Rank Probability Score) shows similar behavior
Filter performances in NEMO
0 50 100 150 200 250 300 350
010
2030
4050
(a) for SSH
Time [days]
rela
tive
RM
SE /
CR
PS [%
]
0 50 100 150 200 250 300 3500
1020
3040
50
(a) for T
Time [days]
rela
tive
RM
SE /
CR
PS [%
]
RMSE_NETFRMSE_LETKFCRPS_NETFCRPS_LETKF
0 50 100 150 200 250 300 350
010
2030
4050
(a) for U
Time [days]
rela
tive
RM
SE /
CR
PS [%
]
0 50 100 150 200 250 300 350
010
2030
4050
(a) for V
Time [days]
rela
tive
RM
SE /
CR
PS [%
]
FIG. 9. Comparison of NETF and LETKF in terms of RMSE (black/gray) and CRPS (red/orange). The lines
represent the field-averaged relative RMSE and CRPS, respectively, for all prognostic variables, i.e., (a) SSH ,
(b) T , (c) U and (d) V , which are defined in Sec. 5.b. The legend in (b) is valid for all panels.
841
842
843
49
Tödter, Kirchgessner, Nerger & Ahrens, MWR 144 (2016) 409 – 427
SSH: Relative error reduction T: Relative error reduction
Nonlinear Ensemble Transform Filter & Smoother
• Smoother reduces filter errors by ~10% • Can be useful as smoothing is cheap to compute
Applying the smoother
day120 240 360 480 600
RM
S er
ror i
n SS
H [m
]
0
0.01
0.02
0.03
0.04
0.05
0.06FilterSmoother lag 30d
RMS errors
lag [days]20 40 60 80 100 120
Ro
10 -5
10 -4
10 -3
10 -2
SSH [m²/day]T [(°C)²/day]U [(m/s)²/day]V [(m/s)²/day]
Roughness
Ro =
Z ✓dRMSE
dt
◆2
dt
• Roughness of estimated trajectory is strongly reduced (smoothed)
Nonlinear Ensemble Transform Filter & Smoother
• Consider relative improvement
Different smoothing impact
lag [days]0 20 40 60 80 100 120
Ri [
%]
0
2
4
6
8
10
12
14
SSHTUV
0 20 40 60 80 100 120lag [days]
0
2
4
6
8
10
12
14
Ri [
%]
sshtuv
L-NETF LETKF
Ri = 100 ·✓1� RMSE
smoother
RMSEfilter
◆
• Similar behavior for ssh (sea surface height) • Distinct for T
! Effect of distinct update schemes (NETF uses observation values for both state and ensemble update)
optimal lag
optimal lag optimal lag
for T
Nonlinear Ensemble Transform Filter & Smoother
Summary
! Nonlinear ensemble transform filter/smoother (NETF/S)
" Update state estimate as particle filter
" Transform ensemble using covariance matrix
! NEMO ocean test case
" NETF filtering performance similar to LETKF
" Slower convergence
" Sensitive on ensemble size
" Successful smoothing
• Dependence on lag distinct for LETKS & NETS (due to different update schemes)
Thank you!