Download - PP KENDA : Contents the Priority Project KENDA · PP KENDA : Contents the Priority Project KENDA (Km-Scale Ensemble-Based Data Assimilation): ... Lucio Torrisi (CNMCA, Italy) Amalia

[email protected], ARPA-SIM, Bologna, 22 November 2010

PP KENDA : Contents

the Priority Project KENDA

(Km-Scale Ensemble-Based Data Assimilation):

an overview of the new assimilation scheme for the COSMO model

Christoph SchraffDeutscher Wetterdienst, Offenbach, Germany

• basic theory

• overview on PP KENDA / preliminary tasks + technical implementation

• results from preliminary experiments

• current work and plans within PP KENDA


after ‘few’ hours, a forecast of convection is a long-term forecast

→ deliver probabilistic (pdf) rather than deterministic forecast

→ need ensemble forecast and data assimilation for convection-permitting NWP

… and Ensemble methods allow to account for the errors of the day

→ Km-scale Ensemble-based Data Assimilation (KENDA):

Local Ensemble Transform Kalman Filter (LETKF, a variant of EnKF family),

because of its relatively low computational costs

KENDA : Why develop Ensemble-Based Data Assimilation ?


Estimation Theory

observation hO → observation error εO

background hB (a-priori info) → background error εB

Assume: - no systematic errors (no bias) - Gaussian distribution of random errors

( ) ( )22

22)(

B

B

O

O hhhh

dh

dJhJ

εε−+−== 0 : of minimum Find

( )

222

22

2

22

2

22

2

111:

OBA

A

BO

OB

BBB

OB

OO

OB

BA hhhhhh

εεεε

εεε

εεε

εεε

+=

−⋅+

+=⋅+

+⋅+

=

error Estimation

estimateBest (in a statistical sense)

( ) ( )2

2

2

2

B

B

O

O hhhhJ

εε−+−= functionCost → measures the distance

to observation and background

Basic theory of data assimilation:estimation theory

( ) ( )

−−

−−=2

2

2

2

expexpB

B

O

O hhhh

εε density yProbabilit

want solution with min. error variance/ maximum likelihood


Estimation Theory

observation hO

→ observation error ε O

background hB

→ background error ε B

Assumptions: - no systematic errors (no bias)- Gaussian distribution of errors

Generalisations

y vector of N observationsR matrix of observation error covariances x vector of M model grid pts * variables (M>>N)B matrix of background error covariances

H ‘observation operator’ : projectingfrom model space into observation space

( )[ ] ( )[ ] [ ] [ ]BBJ xxBxxxyRxy −−+−−= −− 1T1T HH

Find x minimising J . Problems:1. H non-linear → only approx. solution iterativ. 2. Matrix to be inverted too large( )

( )

222

22

2

22

2

111:

OBA

A

BO

BO

OB

BO

OB

BBA

hhh

hhhh

εεεε

εεε

εεε

+=

−⋅+

+=

−⋅+

+=

−−

−

error Estimation

estimateBest (in a statistical sense)

4DVAR: y = ytA→ yt>tA

H(xtA) → H(xt) = H(M(xtA

)) , M : NWP model

measures the distance of the model trajectory to observations within an interval, with t ≥ tA

( ) ( )2

2

2

2

B

B

O

O hhhhJ

εε−+−= functionCost

→ measures the distance to observation and background

Basic theory of data assimilation:estimation theory

no need to compute A, but then need to specify R, B ! usually by statistics (climatology), can be wrong

( ) ( )[ ]BxyRHHRHBxx H−⋅++= 1T11T1 ----BA

→ solution (in model space) : 3DVAR

( ) 11T1 --- HRHBA +=


provides optimal solution to data assimilation problem if – no systematic errors– Gaussian random errors– system linear (M, H ) – (practical issue:) problem small enough to allow computing matrix inversions

Basic theory :Kalman filter

[ ] [ ]bbbTaba HxyKxHxyRHPxx OO −+=−+= −1

( )[ ] 1-1HRH PP

-1 −+= Tba

… and also compute explicitly analysis error covariances (notation: B → Pb )

and then, forecast to the next analysis time (using the forward model M )

na

nbf Mxxx == +1

( )QMPMPP errormodel+== +T

na

nbf

1

analysis step: compute analysis of system state as before (K : Kalman Gain) …


but also need new analysis ensemble xa(i) in order to cycle the system:

– apply Kalman gain to each xb(i) → obs must be artificially perturbed to get reasonable spread (stochastic filter)

– ensemble square root filters, e.g. (L)ETKF :computes analysis ensemble in a ‘deterministic’ way (deterministic filter)

idea : choose ensemble (k members) of initial conditions (model states) such thatensemble spread around analysis mean characterises analysis error Pa ;

then propagate ensemble to next analysis time using the non-linear model, and use resulting forecast ensemble xb(i) to represent forecast error Pb :

Basic theory :Ensemble Kalman filter (EnKF)

( ) bibbTbbb ik xxXXXP −=−= )(,)1( of column th- where

Pb known → can compute Pa, K , analysis mean

for : – non-linear forecast systems (M → NWP model M ) – large problems (large state vector, many obs)

remaining assumptions : – no systematic errors– Gaussian random errors

ax


with model error

→ model error is not accounted for → covariance inflation

Pb = (k – 1) Xb (Xb)T has rank ≤ (k – 1)

→ sampling error for small correlations → localise error covariances→ cannot represent all errors in all degrees of freedom of the global system (model)

• locally, model has much less degrees of freedom → localisation • increase ensemble spread artificially → perturbed obs / covariance inflation

initial (analysis) ensemble spread ‘characterises’ analysis error Pa , but ensemble size is limited, ensemble can only sample but not fully represent errors

Basic theory :Ensemble Kalman filter (EnKF)

ensemble members

area of high probabilitynon-linear

use resulting forecast ensemble xb(i) to represent forecast error Pb :

not represented by ensemble

→ Local ETKF : compute analysis (transform matrix) for each ‘grid point’ separatelyusing only nearby obs


→ ensemble spread is mainly localised over frontal area→ forecast errors assumed in EnKF are mainly localised over frontal area→ observation causes analysis increments over frontal area (see right panel)→ EnKF accounts for errors of the day

Whitaker et al., 2005

Ensemble Kalman filter (EnKF) : a favourable property

use forecast ensemble perturbations Xb(i) to represent forecast error Pb :


for Pb = (k – 1) Xb (Xb)T , J(x) is well-defined in sub-space S spanned by Xb

Basic theory :Ensemble Transform KF (Hunt et al.)

( ) ( ) [ ] [ ])()()( 11xyRxyxxPxxx OO HHJ

TbbTb −−+−−= −−

• regard Xb as linear transformation from k–dim. space S’ onto S ,then for any vector w in S’, Xbw is in S,

• if w is Gaussian random vector with mean 0 and covariance (k – 1) I ,then is Gaussian with mean and cov. (k – 1) Xb (Xb)T bxwXxx bb +=

( ) [ ] [ ])()(1)( 1 wXxyRwXxywww OO bbTbbT HHkJ +−+−+−= −

→ set up cost function in (low-dimensional !) ensemble space

( ) ( ) ( )space obs. in onsperturbati ensemble : of column th-

, , wherebibb

ibibbibibbbbb

i

HHH

yyY

xyyxywYywXx

−=

≡≡=+≈+)(

)()()()() ,

• observation operator H non-linear → approximate :


( ) [ ] [ ]wYyyRwYyywww OO bbTbbTkJ −−−−+−=→ −1)1)(

J(w) : has the form of Kalman Filter cost function

→ solution as above, valid for ensemble mean:

• need analysis ensemble members, with ensemble spread (k – 1) Xa (Xa)T = Pa ,

choose:

( ))()()( iaabbiaaia WwXxXxx ++=+=

[ ]1/2PWWXX a

waaba k )1( −== where,

and in model space : ( )Tbaw

baabba XPXP,wXxx =+=

( ) [ ] ( )[ ]-111 )1( bTbaw

bTbaw

a k YRY I P,yyRYPw O −− +−=−=

• analysis (mean) and analysis error covariance :

note: Pwa is in low-dimensional ensemble space and can be computed directly

→ makes the scheme very efficient

Basic theory :Ensemble Transform KF (Hunt et al.)


→ wanted: ensemble of perturbed analyses, to start next cycle of ensemble forecasts→ analysis mean and each analysis ensemble member is (locally) a linear combination of the forecast ens. members, where those forecast members that match obs well get larger weights; observations reduce spread of ensemble→ explicit solution for minimisation of cost function in ensemble space provides required transform matrix

perturbed forecasts

ensemble mean forecast

(local, inflated) transform

matrixanalysis mean(computed only in ensemble space)

perturbed analyses

( - ) + =( - ) + =( - ) + =( - ) + =( - ) + =

forecast perturbations → flow-dep. background errors

0.9 Pert 1-0.1 Pert 2-0.1 Pert 3-0.1 Pert 4-0.1 Pert 5

(graphics based on a slide of Neil Bowler, UK MetOffice)

Local Ensemble Transform KFLETKF (Hunt et al., 2007)


• implementation following Hunt et al., 2007

• basic idea: do analysis in the space of the ensemble perturbations

– computationally efficient, but also restricts corrections to subspace spanned by the ensemble

– explicit localization (doing separate analysis at every grid point, select only obs in vicinity)

– analysis ensemble members are locally linear combinationsof first guess ensemble members

LETKF for COSMO :summary of method


PP KENDA (km-scale ensemble-based data assimilation) : Overview

Contributions / input by:

Hendrik Reich, Andreas Rhodin, Klaus Stephan, Kathleen Helmert, Uli Blahak (DWD, Germany)

Daniel Leuenberger, Tanja Weusthoff (MeteoSwiss, Switzerland)

Mikhail Tsyrulnikov, Vadim Gorin (HMC, Russia)

Lucio Torrisi (CNMCA, Italy)

Amalia Iriza (NMA, Romania)

[email protected] Deutscher Wetterdienst, D-63067 Offenbach, Germany

• Task 1: General issues in the convective scale (e.g. non-Gaussianity)

• Task 2: Technical implementation of an ensemble DA framework / LETKF

• Task 3: Tackling major scientific issues, tuning, comparison with nudging

• Task 4: Inclusion of additional observations in LETKF

• (Task 5: Development of a Sequential Importance Resampling (SIR) filter)


Task 1: General issues in the convective scale (e.g. non-Gaussianity)

• COSMO Newsletter: M. Tsyrulnikov: Is the Local Ensemble Transform Kalman Filter suitable for operational data assimilation ? (difficulty: assimilate non-local satellite data & achieve good resolution in local analysis)

• COSMO Newsletter: K. Stephan and C. Schraff: The importance of small-scale analysis on the forecasts of COSMO-DE(small-scale analysis can be beneficial, depends on situation, daytime of analysis)

• COSMO Technical Report: D. Leuenberger: Statistical Analysis of high-resolution COSMO

Ensemble forecasts, in view of Data Assimilation(no clear indication that Gaussian assumption of LETKF will be detrimental → COSMO Met Services will (continue to) focus on developing LETKF)


background pdf

small obs error

medium obs error large obs error

observation pdf

shading:pdf after update with stochastic EnKF

shading:pdf after update with deterministic EnKF

Lawson and Hansen, MWR (2004)

solid line: true pdf

General assumption in EnKF: ‘Errors are of Gaussian nature and bias-free’ (prerequisite for ‘optimal’ combination of model forecast and observation)

Effect of non-Gaussianity on EnKF ?


ECMWF

GME

NCEP

UM

fixed perturbations of model physics parameters

temperature at ~10m, +3h (03 UTC)

rlam_heat=0.1

background term: COSMO-DE ensemble forecast perturbations

grandfathers for LBC

Statistical characteristics (COSMO-DE) :background term (EPS)


T@5400 m over N. GermanyT@10 m over Northern Germany (flat)

+3h / +9h+3h / +9h

Statistical characteristics (COSMO-DE) :background term (EPS)

COSMO-DE ensemble forecast perturbations

• results largely determined by physical perturbation technique

• LETKF ensemble will differ form the current set-up of COSMO-DE EPSe.g. Gaussian spread in initial conditions→ need to re-do such evaluations with LETKF ensemble

• for data assimilation, need perturbations that are more Gaussian → physics perturbations: stochastic instead of fixed parameters ?


• basically for verification purposes, COSMO obs operators incl. quality controlwill be implemented in 3DVAR / LETKF environment→ future: hybrid 3DVAR-EnKF approaches in principle applicable to COSMO

• analysis step (LETKF) outside COSMO code→ ensemble of independent COSMO runs up to next analysis time→ separate analysis step code, LETKF included in 3DVAR package of DWD

• standard experimentation system not yet adapted to perform LETKF (but soon)→ stand-alone scripts allow only preliminary LETKF experiments up to now

LETKF (COSMO) :technical implementation


• use in-situ obs (TEMP, AIREP, SYNOP) from GME analysis (sparse density)→ near future: use set of in-situ obs with higher density

• 3-hourly cycles (and only up to 2 days: 7 – 8 Aug. 2009: quiet + convective day)→ (near) future: 1-hourly / 30-min / 15-min cycles

• lateral (and upper) boundary conditions (BC)– ensemble BC from COSMO-SREPS (3 * 4 members),– or deterministic BC from COSMO-EU→ future: ens. BC from global LETKF (GME/ICON)

• ensemble size: 32 (→ near future: ~ 40)

• initial ensemble perturbations reflect global 3DVar-B

• ensemble mean verified against nudging analysiswhich used much more obs

LETKF (COSMO) :set-up of preliminary experiments

COSMO-DE: ∆x = 2.8 km (deep convection explicit, shallow convection param.)

domain size : ~ 1250 x 1150 km


nudging analysis (reference) first guess ensemble spread analysis mean – f.g. mean

analysis mean – nudging ana.

after 1 day of 3-hrly cycling of LETKF

front= area with large spread, i.e. assumingly with large forecast errors

= area with large analysis increments

LETKF (COSMO) , example: zonal wind at lowest model level

= area with differences to nudging analysis


8 Aug. 7 Aug. 8 Aug. 7 Aug.

LETKF, preliminary results:spread compared to RMSE

deterministic BC(with vertical localisation,adaptive R, multiplicative

covariance inflation ρ = 1.15)

free forecastLETKF first guess mean

zonal wind, at ~ 570 hPa pressure at lowest model level

(inner domain average) (inner domain average)

→ obs info isassimilated

RMSE (againstnudging analysis)

first guess meananalysis mean

→ analysis and f.g. have similar RMSE → underdispersive

RMSE

spread




first guess meananalysis mean

ensemble BC(with vertical localisation,adaptive R, ρ = 1.15)

free forecast (perfect BC)LETKF first guess mean



RMSE

spread


→ obs info isassimilated

RMSE (againstnudging analysis)

→ still underdispersive

→ RMSE ana << f.g. for pressure



zonal wind, aircraft obs

time average of RMSE against obs , and spread (7 Aug, 15 UTC – 9 Aug, 0 UTC)

→ larger differences between analysis and first guess at observation locations

zonal wind, radiosondes

→ still underdispersive

(ensemble BC, with vertical localisation, adaptive R, ρ = 1.15)


LETKF, preliminary results:vertical localisation

e-folding decay: 0.3 ln(p)(Gaspari-Cohn)(deterministic BC,

adaptive R, ρ = 1.15)

LETKF first guess mean

no vertical localisationwith vertical localisation




→ better spread / error ratiowith vertical localisation

RMSE

spread


vertical localisation in LETKF: vertically varying linear combination of hydrostatically balanced state need not be hydrostatically balanced

→ balance ? (no (DFI) initialisation applied)

LETKF, preliminary results:balance / noise

→ imbalances slightly increased by vertical localisation

LETKF, deterministic BC:with vertical localisation

no vertical localisationnudging

tps ∂∂from 9 UTC analysis (8 Aug 09)

~ 40 min

LETKF, ensemble BC(with vertical localis.)

nudging

→ by far largest imbalances due to using ensemble BC, not only near lateral BC, but also in inner model domain (larger spread in pressure f.g.)



weight matrices (transform matrices) : determine linear combination

→ diagonal elements >> off-diagonal elements→ analysis increments ’small’

→ do balance analysis increments hydrostatically

normal values for obs errors R R divided by 100(simulating many more obs)

forecastmembers analysis members


hydrostatic balancing applied to analysis increments:


→ initial imbalances strongly reduced (but only in first few timesteps)→ initial imbalances now confined to lateral BC zone

LETKF, ensemble BC:no hydrostatic balancing

with hydrostatic balancingnudging

tps ∂∂from 9 UTC analysis (8 Aug 09)

~ 40 min

tps ∂∂no hydro. balancing with hydro.

balancing

at first timestep


problem: tuning inflation factor ρ takes much time, adaptive procedure preferable

→ (Kalnay et al.:) online estimation of inflation factor(compare ‘observed’ (obs – f.g.) : (y – H(x)) with ‘predicted’ (obs – f.g.) : (R + HTPbH) )

• obs errors / R-matrix probably assumed too small, correction needed

→ compare observed obs error covariance with assumed one (in ensemble space) and correct R if necessary (Li et al., 2009)

• both methods have been tested with reasonable numerical cost and success with the toy model, and have been implemented in the COSMO LETKF

• For COSMO LETKF these methods are currently tested

• lack of spread is (partly) due to model error which is not accounted for so far

• one (simple) method to increase spread is multiplicative covariance inflation:

bb XX ⋅→ ρ (more advanced methods to account for model error (esp. in limited-area models) need to be developed)

LETKF (COSMO) : adaptive methods


LETKF, preliminary results:adaptive covariance inflation ρ

adaptive R : small positive impact, has already been used in all exp. shown



deterministic BC:→ positive impact

RMSE

spread


test: 1 adaptive value for ρ for total domain(LETKF with vertical localisation)

fixed ρ = 1.2adaptive ρ ,

(constant in space)



LETKF, preliminary results:adaptive covariance inflation ρ



RMSE

spread


variable in space(Bonavita et al., QJ10)


ens. BC, adaptive ρ :→ positive impact on

spread-RMSE ratio

ensemble BC:→ negative impact


(constant in space)


• lack of spread: (partly ?) due to model error and limited ensemble size which is not accounted for so far

to account for it: covariance inflation, what is needed ?→ multiplicative Xb → ρ · Xb (tuning, or adaptive (y – H(x) ~ R + HTPbH))

→ additive (stochastic physics , statistical 3DVAR-B , etc. )

• model bias

• localisation (multi-scale data assimilation,2 successive LETKF steps with different obs / localisation ?)

• update frequency

• noise control: – how to deal with perturbed lateral (upper ?) BC (distort implicit error covariances in filter → limit use of obs ?)

– hydrostatic imbalance by vertical localisation

– digital filter initialisation ?

• non-linear aspects, convection initiation (latent heat nudging ?)( → CH project submitted: OSSE, investigate / reduce spin-up (‘outer loop’) )

end of 2011: (‘pre-operational’) LETKF suite

LETKF (km-scale COSMO) : scientific issues / refinement


• stochastic physics (Palmer et al., 2009) (by Lucio Torrisi, available Jan. 2011)

• estimation and modelling of model-errors(by M. Tsyrulnikov, V. Gorin (Russia), for 2 years, start in June 2010)

1. develop an objective estimation technique for model (tendency) errors(not to be confused with forecast errors), and set upstochastic model (parameterisation) for model error : e = u * M(x) + eadd

involves stochastic physics ( u * M(x) ) and additive components eadd

and includes multi-variate and spatio-temporal aspects

– build statistics of forecast tendencies minus observation tendencies d(using pairs of lagged obs (increments) at nearly the same location)cov(d) = ( cov(u) M(xi) M(xi) + cov(δM(x)) + cov(eadd) ) * ∆t 2 + 2 R

where δM(x) = error in the model tendency due to the analysis error, cov(δM(x)) estimated by sample covariance of the ens. tendencies

Obs: p, T, u,v, q: Synop, ship, buoy; aircraft; wind profiler (?); radar (?)

2. estimate parameters for model error by fitting to statistics

3. develop a model-error generator and embed it in the COSMO code

4. validate: apply MEM to COSMO forecasts and check resulting EPS spread



Statistics: 2 months in winter and in summer. T, u, v fields, 1000-100 hPa. SYNOP & TEMP obs. on COSMO-RU domain


Conditional expectation and variance of ME as a function of model 12-h tendency (T_500 summer)

-1.5-1

-0.50

0.51

1.52

2.5

-10 -5 0 5 10

Temperature Tendency, K

Tem

pera

ture

err

or,

K

Mean:

M_appr:

STD:

S_appr:

the upper curves imply a non-zero var(u), the lower curves E[u].

For most cases, both conditional mean and conditional variance are present.


• investigate LETKF in Observing System Simulation Experiments (OSSE)

→ project plan submitted by MeteoSwiss ,for 2 years, start May 2011

– apply LETKF to idealized convective weather systems (at the convection permitting scale) , tune LETKF settings (localization, covariance inflation)

– quantify non-Gaussianity and spin-up time in the evolving LETKF ensemble during the assimilation of a convective storm

– develop ways to improve Gaussianity of errors and reduce spin-up time of the LETKF → test “running in place” (Kalnay and Yang, 2010)

and “outer loop” (Yang and Kalnay, 2010) algorithms

LETKF (COSMO) : scientific issues / refinement

• MeteoSwiss: analysis for deterministic forecast with 1-km resolution

– use Kalman Gain of LETKF, interpolate to 1 km, and apply to obs increments to obtain analysis increments, or do full LETKF at 1 km resolution

– start Oct. 10: first compare approaches with toy model, later with COSMO


[ ] ( ) ( ) 11)( −−

=−+= RYPXK,xyKxx O Tbaw

bBBA LH

Kalman gain / analysis increments not optimal, if deterministic background xB (strongly) deviates from ensemble mean background

Analysis for a deterministic forecast run :use Kalman Gain K of analysis mean

( ) bTbaw k YRY I P 1)1( −+−=

L : interpolation of analysis increments from grid of LETKF ensemble to (possibly finer) grid of deterministic run

deterministic

( ) 1

,,

−RY

PXTb

aw

b

ensemble


• radar : radial velocity and (3-D) reflectivity

observation operators (2 PhD’s, started summer 2010, DWD funding) .

– implement full, sophisticated observation operators

– then develop different approximations and test their validity (both looking at the simulated obs and doing assimilation experiments) in order to obtain sufficiently accurate and efficient operators

Particular issues for use in LETKF: obs error variances and correlations,superobbing, thinning,localisation

LETKF, Task 4: inclusion of additional observations

• ground-based GNSS slant path delay (after 2011, PhD DWD / Uni Reading)

– implement non-local obs operator in parallel model environment

– test and possibly compare with using GNSS data in the form of IWV or of tomographic refractivity profiles

Particular issue: localisation for (vertically and horizontally) non-local obs


• cloud information based on satellite and conventional data

→ DWD: Eumetsat fellowship (plus resources from regular staff)

– derive incomplete analysis of cloud top + cloud base, using conventional obs (synop, radiosonde, ceilometer) and NWC-SAF cloud products from SEVIRI

– use obs increments of cloud or cloud top / base height or derived humidity

– use SEVIRI brightness temperature directly in LETKF in cloudy (+ cloud-free) conditions, in view of improving the horizontal distribution of cloud and the height of its top

– compare approaches

Particular issues: non-linear observation operators, non-Gaussian distribution of observation increments

LETKF, Task 4: inclusion of additional observations


KENDA

thank you for your attention


• „No cost smoother“ (Kalnay and Yang, QJRS, submitted)

– Apply weights from time t+1 at time t

• „Running in place“– Use observations more than once by iterating several times over

same assimilation window using the no cost smoother until „convergence“

• „Outer loop“ in LETKF (Yang and Kalnay, MWR, submitted)

– Bring analysis mean closer to observations by iterative integration of ensemble mean from time n-1 to n (re-centering)

– Advance to next assimilation time by integration of whole ensemble to time n+1

• Have proven to improve LETKF in presence of nonlinearity / non-Gaussianity in Lorenz model

LETKF (COSMO) : method to deal with non-Gaussianity

Download - PP KENDA : Contents the Priority Project KENDA · PP KENDA : Contents the Priority Project KENDA (Km-Scale Ensemble-Based Data Assimilation): ... Lucio Torrisi (CNMCA, Italy) Amalia

Top Related