hybrid (3d-var /etkf) data assimilation system

Meral Demirtas National Center for Atmospheric Research

Hybrid (3D-VAR /ETKF) Data Assimilation System

WRF Tutorial, WRF-Data Assimilation

21st July 2009, Boulder, Colorado

An outline of this presentation

Why do we need a hybrid data assimilation system?

What are the basic ingredients of a hybrid system?

How have we implemented hybrid (3DVAR -ETKF) system at the Data Assimilation Testbed Center (DATC)?

What we have found: Highlights of preliminary results

Summary and conclusions

An introduction for the hybrid practice session

Why do we need a hybrid system?

• The WRF 3D-VAR system uses only climatological (static) background error covariances.

• Flow-dependent covariance through ensemble is needed.

• Hybrid combines climatological and flow-dependent background error covariances.

• It can be adapted to an existing 3D-VAR system.

• Hybrid can be robust for small size ensembles.

What are the basic ingredients of a hybrid system?

1. Ensemble forecasts: WRF-ensemble forecasts

2. A mechanism to update ensemble perturbations: Ensemble Transform Kalman Filter (ETKF)

3. A data assimilation system: WRF 3D-VAR

It sounds simple…. :-)

Ensembles to address uncertainties in the initial state

Ensemble mean

observed (idealized!)

Ensemble members

Ensemble members

x f

x1f

xnf

time

Initial state

Ensemble Formulation Basics

Assume the following ensemble forecasts:

Ensemble mean:

Ensemble perturbations:

Ensemble perturbations in vector form:

X f =(x1f ,x2

f ,x3f ,.....xN

f )

€

x f =1

Nxn

f

i=1

N

∑

€

δxnf = xn

f − x f

€

n =1,N

€

δX f = (δx1f ,δx2

f ,δx3f ,....,δxN

f )

How to update ensemble perturbations?

ETKF technique updates ensemble perturbations by rescaling innovations with a transformation matrix (Wang and Bishop 2003).

Transformation matrix (solved by Kalman Filter Theory)

Τ =C(Γ + Ι)−1/2 CT

C: is the column matrix that contains the orthonormal eigenvectors

: is the diagonal matrix that contains the eigenvaluesΓ

Bishop et al. (2001)

I: is the identity matrix

xa =xfT

How to improve the under estimation of the analysis-error variance?

The background-error variance estimated from the spread of ensembles should be consistent with the background error-variance estimated from the differences between the ensemble mean and observations. Wang and Bishop (2003) has introduced an adaptive inflation factor, ∏, to ameliorate this problem:

P i = c1 ,c2,...,ci

where

%di = R- 1/2 yi - Hxi

b( )

xi =xifTi i∏

ci =

tr %di%di

T - Nobs

tr(G)

Schematic illustration of the ETKF technique

Courtesy of Bowler et al. (2008)

Pros and Cons of ETKF Technique

• Desirable aspects:– ETKF is fast (computations are done in model ensemble

perturbation subspace).– It is suitable for generating ensemble initial conditions.– It updates initial condition perturbations.

• Less desirable aspects:– ETKF does not localize, therefore it does not represent

sampling error efficiently. It may need very high inflation factors.

Hybrid: Combine 3D-VAR and ETKF

Flow-dependent covariance through ensembles. Coupling wind, temperature and moisture fields. Hybrid can be more robust for small size

ensembles. It can be adapted to an existing 3D-VAR system. It is less computationally expensive compared to

other ensemble filters.

The hybrid formulation….Ensemble covariance is implemented into the 3D-VAR cost function via extended control variables:

J(x1' ,α) =β1

12

x1'T B−1x1

' + β2

12αTC−1α +

12(yo' −Hx')T R−1(yo' −Hx')

x' =x1' + (αk oxk

e)k=1

K

∑

3D-VAR incrementx1'

x' Total increment including hybrid

β1 Weighting coefficient for static 3D-VAR covariance

β2 Weighting coefficient for ensemble covariance α Extended control variable

C: correlation matrix for ensemble covariance localization

(Wang et. al. 2008)

1

β1

+1β2

=1Conserving total variance requires:

Ensemble covariance horizontal localization is done through recursive filters. Preconditioning designed as:

x '1=U1v1 U1≈B1/2

α=U2 v2U2 ≈C1/2

(Wang et. al. 2008)

Hybrid Horizontal Localization

C is the ensemble correlation matrix which defines ensemble covariance localization.

where

where

• 3D-VAR part

• Ensemble part

B is the 3D-VAR static covariance matrix.

Hybrid Vertical Localization• Spurious sampling error are not confined to horizontal error

correlations, it affects vertical too.

• Hybrid alpha control variables can be made 3D to alleviate

spurious vertical correlations.

• Two approaches are considered for specifying vertical localization function:

i. Empirical function (as in horizontal).

ii. Use vertical background error covariances to define

localization.

• Initial studies use “empirical function (as in horizontal)”.

• An EOF decomposition of the vertical component of the localization matrix is performed to reduce size of alpha CV.

Courtesy of Dale Barker

Empirical Vertical Covariance Localization

Apply Gaussian Vertical Covariance Localization function:

ρ(k − kc ) = exp − k − kc( )2

/ Lc2⎡

⎣⎤⎦

Example 1:Constant Localization Scale

Example 2:Vertically-Dependent Localization Scale

Lc =10 Lc =20kc / 41Courtesy of Dale Barker

Covariance Localization DecompositionExample: Gaussian Localizationwith variable localization scale:

ρ(k − kc ) = exp − k − kc( )2

/ Lc2⎡

⎣⎤⎦

Lc =20kc / 41

Eigenvectors Eigenvalues

75% data compression via use of EOFs for covariance localization.Courtesy of Dale Barker

Vertical cross-section of analysis increments (single-observation based )

No vertical localization applied Vertical localization applied

Courtesy of Dale Barker

How have we implemented hybrid (3DVAR -ETKF) system at the (DATC)?

WRF Ensemble:

M1

M2

M3

M4

M5

….

Mn

Compute: ensemble mean

Compute: Ensemble perturbations (ep2): u,v,t,ps,q, mean, and std_dev

WRF-VAR (VERIFY): ob.etkf ensemble

ETKF: Update ens perturbations

Update: ens. initial conditions

Update: ens. boundary conditions

WRF-VAR (QC-OBS): filtered_ob.ascii

WRF ensemble run for the next cycle

cycling

WRF-VAR (hybrid)

Update: ensemble mean

WRF deterministic forecast run

(Demirtas et al. 2009)

A few notes on hybrid settings

• alpha_corr_scale=1500km (Default)

• je_factor ( )=2.0

• jb_factor ( )=je_factor/( je_factor -1 )=2.0• alphacv_method=2 (ensemble perturbations on

model space)

• ensdim_alpha=10 (ensemble size)

β1

β2

N.B. Conservation of total variance requires:1

β1

+1β2

=1

The DATC Hybrid System Application Experiment Set-up

• Ensemble size: 10

• Test Period: 15th August - 15th September 2007

• Cycle frequency: 3 hours

• Observations: GTS conventional observations

• Deterministic ICs/BCs: Down-scaled GFS forecasts

• Ensemble ICs/BCs: Produced by adding spatially correlated Gaussian noise to GFS forecasts (Torn et al. 2006).

(WRF-VAR and some additional tools.)

• Horizontal resolution: 45km

• Number of vertical levels: 57

• Model top: 50 hPa

(For details see: Demirtas et al. 2009)

What we have found: Highlights of preliminary results

Ensemble spread: 500 hPa height (m) std. dev.

WRF t+3 valid at 2007081900

Modest inflations factors used Higher inflations factors used

Ensemble Mean and Std. Deviation (spread)

Inflation Factors (from 3-hourly cycling)

Stable, no-smoothing has been applied yet.

Hybrid gives better RMSE scores for wind compared to 3D-VAR.

Hybrid gives better RMSE scores for wind compared to 3D-VAR.

Hybrid gives better RMSE scores for wind compared to 3D-VAR.

ECMWF analysis data (T106) used.

ECMWF analysis data (T106) used.

Hybrid gives better RMSE scores for wind compared to 3D-VAR.

Cost function: 3DVAR and Hybrid

Cost function is smaller in hybrid.

Preliminary results from JME applications

(snapshots)

Joint Mesoscale Ensemble (JME) Applications 10-m wind speed ensemble standard

deviation (spread)• 10 to 20 WRF physics configurations

• Capability for WRF-Var to update mean and/or individual members

• Capability for ETKF perturbations

• Lateral boundary conditions from global ensemble (GFS)

• Research on multi-parameter and stochastic approaches

• WRF-Var used to compute innovations.

Courtesy of Josh Hacker

Inflation Factors Generated by JME

Courtesy of Soyoung Ha Twice daily cycling at 00 and 12 UTC

Referred referencesBishop, C. H., B. J. Etherton, S. J. Majumdar, 2001: Adaptive sampling with the ensemble transform Kalman filter. Part I: Theoretical aspects. Mon. Weather Rev., 129, 420–436.

Bowler, N. E., A. Arribas, K. R. Mylne, K. B. Robertson, S. E. Beare, 2008: The MOGREPS short-range ensemble prediction system. Q. J. R. Meteorol. Soc. 134, 703–722.

Demirtas, M., D. Barker, Y. Chen, J. Hacker, X-Y. Huang, C. Snyder, and X. Wang, 2009: A Hybrid Data Assimilation System (Ensemble Transform Kalman Filter and WRF-VAR) Based Retrospective Tests With Real Observations. Preprints, the AMS 23rd WAF/19th NWP Conference, Omaha, Nebraska.

Torn, R. D., G. J. Hakim, and C. Snyder, 2006: Boundary conditions for limited area ensemble Kalman filters. Mon. Wea. Rev., 134, 2490-2502.

Wang, X., and C. H. Bishop, 2003: A comparison of breeding and ensemble transform Kalman filter ensemble forecast schemes. J. Atmos. Sci., 60, 1140-1158.

Wang, X., D. Barker, C. Snyder, T. M. Hamill, 2008: A hybrid ETKF-3DVAR data assimilation scheme for the WRF model. Part I: observing system simulation experiment. Mon. Wea. Rev., 136, 5116-5131.

What is on the menu for the hybrid practice session

Computation:• Computing ensemble mean.

• Extracting ensemble perturbations (EP).

• Running WRF-VAR in “hybrid” mode.

• Displaying results for: ens_mean, std_dev, ensemble perturbations, hybrid increments, cost function and, etc.

• If time permits, tailor your own test by changing hybrid settings; testing different values of “je_factor” and “alpha_corr_scale” parameters.

Scripts to use:• Some NCL scripts to display results.

Brief information for the chosen caseEnsemble size: 10

Domain info:• time_step=240,

• e_we=122,

• e_sn=110,

• e_vert=42,

• dx=45000,

• dy=45000,

Input data provided (courtesy of JME Group):• WRF ensemble forecasts valid at 2006102800

• Observation data (ob.ascii) for 2006102800

• 3D-VAR “be.dat” file

Thanks for attending…..