insights on observation residual approaches for model and ...introduction residual estimation of...
TRANSCRIPT
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Insights on observation residual approaches formodel and observation error estimation
Ricardo Todling
Global Modeling and Assimilation Office, NASA
10th Adjoint Workshop1-5 June 2015
Roanoke, West Virginia
In collaboration with: Yannick Tremolet
Discussions with: S. E. Cohn, D. P. Dee, M. Fisher, & M. Leutbecher
Ricardo Todling Insights on error estimation 1 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Outline
1 Introduction
2 Residual Estimation of System Error
3 Curiosities of Estimates from Residual Statistics
4 What do we get from 4d-Var Residual Diagnostics?
5 Recast Sequential Approach for Q into Variational Language
6 Closing Remarks
Ricardo Todling Insights on error estimation 2 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Filtering Approach
Introduction
Ricardo Todling Insights on error estimation 3 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Filtering Approach
Residual Statistics from Sequential Filtering Approach
Observation-minus-Background (OmB) residiuals
dbo = yo −Hxb
OmB error covariance matrix
< dbo(dbo)T >= HBHT + R
Sample error covariances of OmB residuals have traditionally beenused to estimate and model time-independent background errorcovariances (Schlatter 1974, Hollingsworth & Lonnberg 1986, Dee& da Silva 1999, Franke 1999, and others, going back to the workof T. Kailath in the 70s).
The expression above holds independently of optimality (though itrelies on assumptions about background and observation errors).
Ricardo Todling Insights on error estimation 4 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Filtering Approach
Residual Statistics from Sequential Filtering Approach
Other available residuals:
dao = yo −Hxa dab = Hxb −Hxa
which lead to the following cross-covariances:
< dao(dbo)T > = R + O(∆K)
< dab(dbo)T > = HBHT + O(∆K)
< dab(dao)T > = HAHT + O(∆K)
where ∆K is the deviation of the sequential filter gain from optimality.
Ricardo Todling Insights on error estimation 5 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Filtering Approach
General Remarks on Residual Statistics
Only under the assumption of optimality, ∆K = 0, thecross-covariances become covariances and the expressionsabove can be used to estimate observation, background, andanalysis error without concerns (Desroziers et al. 2005).
The validity of replacing the expectation operator, < • >,with the typical time average operator might not always bejustifiable.Derivation of residual cross-covariances corresponding tothose of the variational approach must be inferred carefully totry minimizing, if not possibly avoiding altogether, introducingtime correlations.
Ricardo Todling Insights on error estimation 6 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Filtering Approach
Residual Estimation of System Error
Ricardo Todling Insights on error estimation 7 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Smoothing ApproachIllustration: Application to L96
Residual Statistics from Sequential Smoothing Approach
Residuals from a sequential lag-1 smoother can be used to derive anestimate of the model error covariance∗:
dso = yo −HMxs dsa = Hxa −HMxs
which lead to the following cross-covariances:
< dso(dbo)T > = HQHT + R + O(∆G)
< dsa(dso)T > = HQHT + O(∆K) + O(∆G)
where ∆G is the deviation of the sequential smoother gain fromoptimality.
∗ see Todling (2014; QJRMS).
Ricardo Todling Insights on error estimation 8 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Smoothing ApproachIllustration: Application to L96
Illustration in L96
Examples for simple linear and nonlinear models serve as illustration ofthe method. For L96, with stochastically random Gaussian noise withcovariance of the form below . . .
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
site
site
True Q covariance
−0.1
−0.05
0
0.05
0.1
0.15
0.2
Specified model error covariance
Ricardo Todling Insights on error estimation 9 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics from Sequential Smoothing ApproachIllustration: Application to L96
Illustration in L96Residual statistics from lag-1 smoothers implemented for two assimilationmethodologies, namely the EKF and an adaptive OI, is shown to obtainreasonable estimates of Q.
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
site
site
sto: Regularised EKF Q estimate
−0.1
−0.05
0
0.05
0.1
0.15
0.2
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
site
site
sto: AOI Q estimate
−0.1
−0.05
0
0.05
0.1
0.15
0.2
EKF AOI
From Todling (2014; QJRMS)
Ricardo Todling Insights on error estimation 10 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs
Curiosities of Estimates from Residual Statistics
Ricardo Todling Insights on error estimation 11 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs
Model or Observation Error?Still examining the L96 application, when the residual diagnostic is usedto estimate R ≈< dao(dbo)T >, the derived estimates have a considerablesignature of the model error.
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
site
site
sto: Regularised EKF R estimate
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
5 10 15 20 25 30 35 40
5
10
15
20
25
30
35
40
site
site
sto: AOI R estimate
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
EKF AOIFrom Todling (2015; QJRMS)
Ricardo Todling Insights on error estimation 12 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs
Actual vs Prescribed OmB Residual Statistics: Original
t race(v*v)/ p ( 0.4761)trace(HPH+ R)/ p ( 0.4984)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 100 200 300 400 500 600 700 800
sto: Regularised EKF MC50
cycle
actu
al v
s p
resc
rib
ed v
aria
nce t race(v*v)/ p ( 0.4933)
trace(HPH+ R)/ p ( 0.4935)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 100 200 300 400 500 600 700 800
sto: Adapt ive OI MC50
actu
al v
s p
resc
rib
ed
var
ian
ce
EKF AOI
Ricardo Todling Insights on error estimation 13 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs
Actual vs Prescribed OmB Residual Statistics: Second Pass
t race(v*v)/ p ( 0.4922)trace(HPH+ R)/ p ( 0.4929)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 100 200 300 400 500 600 700 800
sto: Regularised EKF MC50
cycle
actu
al v
s p
resc
rib
ed v
aria
nce t race(v*v)/ p ( 0.5372)
trace(HPH+ R)/ p ( 0.5057)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 100 200 300 400 500 600 700 800
sto: Adapt ive OI MC50
cycle
actu
al v
s p
resc
rib
ed v
aria
nce
EKF AOI
Ricardo Todling Insights on error estimation 14 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs
Model or Observation Error?Not knowing any better - that is, that the error in the L96 application isdue to errors in the model - using the R estimate as a “correction” to thespecified observation error covariance leads to undesirable consequences.For example, re-estimation of R brings new estimates further away fromtrue value.
diag(R)diag(Re)diag(Reff)
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0 5 10 15 20 25 30 35 40
sto: Regularised EKF MC50
diag(R)diag(Re)diag(Reff)
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0 5 10 15 20 25 30 35 40
sto: Adapt ive OI MC50
EKF AOI
From Todling (2015; QJRMS)
Ricardo Todling Insights on error estimation 15 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs
Lesson from OSSEs
Somewhat analogous to what we just seen are various estimates ofinter-channel correlations obtained from operational systems and OSSE’sbuild off these systems. Funny enough, OSSE residual statistics mightsuggest observations to be largely correlated when in fact they are not.Examples from GEOS-5 are given below.
0 20 40 60 80 100 1200
20
40
60
80
100
120GMAO OPS: AIRS AQUA
?1.0
?0.8
?0.6
?0.4
?0.2
0.0
0.2
0.4
0.6
0.8
1.0
0 20 40 60 80 100 1200
20
40
60
80
100
120GMAO (perfect) OSSE: AIRS AQUA
?1.0
?0.8
?0.6
?0.4
?0.2
0.0
0.2
0.4
0.6
0.8
1.0
0 20 40 60 80 100 1200
20
40
60
80
100
120GMAO OSSE: AIRS AQUA
?1.0
?0.8
?0.6
?0.4
?0.2
0.0
0.2
0.4
0.6
0.8
1.0
From GEOS-5 OPS From early OSSE (perfect obs) From tuned OSSE
Inspired by comment make by Anna Shlayeva at the Workshop on Correlated Obs. Errors, Reading U.K., 2014.OSSE results from exps in Errico et al. (2013; QJRMS)Results here from Todling (2015; QJRMS)
Ricardo Todling Insights on error estimation 16 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
What do we get from 4d-Var Residual Diagnostics?
Ricardo Todling Insights on error estimation 17 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
4d-ResidualsConsider SC 4d-Var in its 4d-PSAS form:
J(δx0) =1
2δxT0 B
−1δx0 +1
2(db −Hδx0)TR−1(db −Hδx0) ,
For
db ≡ [(db0)T (db1)T · · · (dbK )T ]T
da ≡ [(da0|0)T (da1|0)T · · · (daK |0)T ]T
dba ≡ db − da
it is simple to show that
< dba(db)T > = HBHT (HBH + R)−1 < db(db)T >”opt”
= HBHT .
Ricardo Todling Insights on error estimation 18 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
4d-ResidualsWhen the evolution of forecast errors within the var-window is taken into account(i.e., considering how model error also propagates in the window), we really have
< db(db)T >= HPf0|−1H
T + R
where R is given by
R`,m ≡ R`δ`,m + H`|−1
`−1∑i=0
m−1∑j=0
M`,`−i|−1Q`−i,m−jMTm,m−j|−1
HTm|−1 ,
When B = Pf0|1, somewhat unsurprisingly, one finds that
< da(db)T >≈ R
This means:The cross covariance of “OmA” and “OmB” from 4d-Var give an estimate of aneffective observation error covariance.Indeed, only at initial time do we the sought out observation error covariance,i.e., < da0(db0)T >≈ R.
Ricardo Todling Insights on error estimation 19 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Brief argumentPreliminary estimation in the IFS
Recast Sequential Approach for Q into Variational Language
Ricardo Todling Insights on error estimation 20 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Brief argumentPreliminary estimation in the IFS
Residual Q estimation: sequential to variationalMotivated by the sequential approach, we consider two loosely connected 4d-Varproblems:
one being double the window size of the otherthink of the short-window as the filterthink of the long-window as the smootherfurthermore, each long window cycle is started from the short-window analysis
and derive expressions that relate the two problems to allow extracting information onthe model error covariance:
(< wI1
(dbI1)T >
)`,m
opt=
−1∑i=ks
−1∑j=ks
H`M`,`−i|ks−1Q`−i,m−jMTm,m−j|ks−1H
Tm + O(∆M`,m)
< wI1(dbI1
)T >opt= HQHT
wheredbI1
is the OmB residual in the first half of the short or long-window problems
wI1is an incremental residual differencing the OmA of the short-window with
that of the long-window problem within the first half of the long-window
Ricardo Todling Insights on error estimation 21 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Brief argumentPreliminary estimation in the IFS
Residual Q estimation: sequential to variational
T 0 0.2 0.4 0.6 0.8 1 1.2 1.4
1000
900
850
700
500
400
300
RH errors
Northern Hemisphere
sigbsigosigb2siga06siga12sigq
Q 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6
1000
850
700
500
250
100
50
30
10
Tv errors (K)
Northern Hemisphere
sigbsigosigb2siga06siga12sigq
U 0 0.5 1 1.5 2 2.5 3
1000
850
700
500
250
100
50
30
10
U errors (m/s)
Northern Hemisphere
sigbsigosigb2siga06siga12sigq
V 0 0.5 1 1.5 2 2.5 3
1000
850
700
500
250
100
50
30
10
V errors (m/s)
Northern Hemisphere
sigbsigosigb2siga06siga12sigq
Ricardo Todling Insights on error estimation 22 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Closing Remarks
Ricardo Todling Insights on error estimation 23 / 24
IntroductionResidual Estimation of System Error
Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?
Recast Sequential Approach for Q into Variational LanguageClosing Remarks
Closing Remarks
Many of the insights on residual diagnostics presented here havebeen appreciated by others - largely in this audience.
However, I hope to have provided a few illustrations highlightingwhat we know from theory: that in actuality, background, modeland observations errors are inseparable.
It is possible to estimate model error covariances from residualstatistics using smoother ideas, but the entanglement withbackground and observations errors, particularly in variationalapproaches, make it really hard to obtain conclusive results.
Remark related to words on “Historical Overview”:
Recently, Grewal & Andrews (2010: IEEE Control Systems Magazine, 69-78)provides a nice review of the use of Kalman filtering in Aerospace. It seemsunfortunate, though, that these authors are not aware of the earlier review ofMcGee & Schmidt (1985: NASA TM 86847).
Ricardo Todling Insights on error estimation 24 / 24