lecture 1 basics on data assimilation - earth online -...
TRANSCRIPT
Day 5 Lecture 1 Module name - Basics on Data Assimilation 1
Lecture 1Basics on Data Assimilation
H. ElbernRhenish Institute for Environmental Research at the
University of Cologneand
Institute for Chemistry and dynamics of the Geosphere-2: Troposphere (ICG-2), Research Centre Jülich, Germany
Day 5 Lecture 1 Module name - Basics on Data Assimilation 2
Objective of this lectureUnderstanding of• the fundamental objectives of data
assimilation and inverse modelling• basic theoretical ideas of data
assimilation• numerical implementation measures for
spatio-temporal assimilation algorithms
Day 5 Lecture 1 Module name - Basics on Data Assimilation 3
General textbook literature for data assimilation• Daley, R., Atmospheric Data Analysis, Cambridge University Press, pp. 457, 1991.
– Well established connection between statistics and practical data assimilation– Meanwhile behind cutting edge operational data assimilation implementations
• Bennet, A., Inverse Methods in Physical Oceanography, Cambridge University Press, pp. 346, 1992.
• Bennet, A., Inverse Modeling of the Ocean and Atmosphere, Cambridge University Press, pp. 234, 2002.
– Mathematically very sound, with focus on oceangraphy– emphasis on science, rather than operational usage
• Geir Evensen Data Assimilation: The Ensemble Kalman Filter, Springer, 2006
• Kalnay, E., Atmospheric Modeling, Data Assimilation and Predictability, Cambridge University Press, pp. 341, 2003, Chap.
– Well presented pedagogical introduction to data assimilation theory
• Thiebaux, H.J. and Pedder, M.A. Spatial Objective Analysis with applications in atmospheric science, Academic Press1987.
– Mathematically rigorous formalism with Optimum Interpolation as central algorithm
Additionally, a collection of overview papers:• Special Issue: J. Met. Soc. Japan, Vol. 75, No. 1B, pages 1-138, 1997.• ECMWF tutorials: www.ecmwf.int
Day 5 Lecture 1 Module name - Basics on Data Assimilation 4
Underlying mathematics(advanced level):
• Jazwinski, A., Stochastic Processes and Filtering Theory, Academic Press,San Diego, pp. 376, 1970.
• Marchuk, G., Numerical solutions of the problems of the problems of thedynamics of the atmosphere and ocean, Gidrometeoizadat, Leningrad, 1974.
• Courant, R. and Hilbert, D. Methods of mathematical physics, Volume 1, New York: Interscience, 1953.
• Courant, R. and Hilbert, D. Methods of mathematical physics, Partial differentialequations, Volume 2, New York: Interscience, 1962.
• Lions, J., Optimal control of systems governed by partial differential equations, Spinger Verlag, Berlin, 1971.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 5
TerminologyInverse ModellingThe inverse modelling problem consists of using
the actual result of some measurements to infer the values of the parameters that characterize the system.
A. Tarantola (2005)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 6
Objective of atmospheric data assimilation (1)
The ambitious and elusive goal of dataassimilation is to provide a dynamicallyconsistent motion picture of theatmosphere and oceans, in three spacedimensions, with known error bars.
M. Ghil and P. Malanotte-Rizzoli (1991)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 7
Objective of atmospheric data assimilation (2)• "is to produce a regular,
physically consistent four dimensionalrepresentation of the state of the atmosphere
• from a heterogeneousarray of in situ and remote instruments
• which sample imperfectly and irregularly in space and time.
Data assimilation • extracts the signal from
noisy observations (filtering)
• interpolates in space and time (interpolation) and
• reconstructs state variables that are not sampled by the observation network (completion).“ (Daley, 1997)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 8
This lecture introduces:Basic theoretical ideas of data
assimilation
1. three basic sources of information2. combination of information sources3. the basic ideas of 4D-var and Kalman-filter4. the limiting assumptions and a look forward
Day 5 Lecture 1 Module name - Basics on Data Assimilation 9
y
a
b { |1
|0 x
y=ax+b
Data assimilation has much of curve fitting:e.g. quadratic optimisation
Day 5 Lecture 1 Module name - Basics on Data Assimilation 10
Characteristics in data assimilation,in contrast to remote sensing retrievals
• high dimensional problem: dim (x) > 105
• highly underdetermined system (few observations with respect to model freedom: dim(x)>>dim(y) )
• regionally/locally highly nonlinear dynamics• constraints by physical laws are insufficient
Day 5 Lecture 1 Module name - Basics on Data Assimilation 11
Advanced data assimilation is an application of the principles of Data Analysis.
(As in satellite retrievals) we strive for estimating a latent, not apparent parameter set x.
We dispose of 1. indirect information on the state and processes in terms of
manifest, though insufficient or inappropriate data y, and2. a deterministic model M connecting x and y, in some way.
Bayes’ rule gives access to the probability of state x, given y and M
Day 5 Lecture 1 Module name - Basics on Data Assimilation 12
inversion
radiance dataasimilation
dataretrieval
DA in the satellite data application chainlevel 0
detector tensions
level 1spectra
level 2geolocated
geophysical parameters
level 3 geophysical parameters
on regular grids
level 4 “user” applications
priordata
geophysicalmodels
Day 5 Lecture 1 Module name - Basics on Data Assimilation 13
PredictabilityWhich credibility can the prediction claim?
Compute probability density function
(in some way by “optimal perturbations”)
ObservabilityGiven an observation set,“how unambiguous” can
the state estimate be?compute (some norm of)analysis error covariance
matrix
SensitivityHow probable/stable is the simulation/state ?
compute leading singular vectors
discontinuities
Data AssimilationInverse Modelling
estimate most probable stateparameter values
ill p
osed
ness
whi
ch o
bser
vatio
ns?
mos
t pro
babl
epr
edic
tion
optimal perturbations
DA and related algorithms: How can dynamic system understanding be
assessed?
Day 5 Lecture 1 Module name - Basics on Data Assimilation 14
Ad hoc (empirical) approaches (1)Interpolation
Day 5 Lecture 1 Module name - Basics on Data Assimilation 15
Ad hoc (empirical) approaches (2)Nudging
Caution: prone to excitement of artificial modes
Day 5 Lecture 1 Module name - Basics on Data Assimilation 16
Optimality criteria: Which property can be attributed to our analysis result?
(Need for quantification)• maximum likelyhood:
– maximum of probability density function• minimal variance: l2 norm
– parameters optimal, for which analysis error spread is minimal (for Gaussian/normal and log-normal error distributions), Best Linear Unbiased Estimate (BLUE)
• minimax norm (discrete cases)• maximum entropy
Day 5 Lecture 1 Module name - Basics on Data Assimilation 17
Data assimilation: Synergy of Information sourcesexample: 2 data sets with Gaussian distribution
(here: minimal variance= maximum likelihood)
a priori (=prediction or climatology) observation
Bayes’ rule:
xa
Analysis (=estimation) BLUEBLUEBest Linear Unbiased Estimate
xb yoxaxb yoxb
Example: inconsistent data
Day 5 Lecture 1 Module name - Basics on Data Assimilation 18
Optimal Interpolation
Notation: Ide, K., P. Courtier, M. Ghil, and A. Lorenc, Unified notation for data assimilation: operational sequential and variational, J. Met. Soc. Jap., 75, 181--189, 1997.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 19
Kalman filter: basic equations
Day 5 Lecture 1 Module name - Basics on Data Assimilation 20
4D-var
Kalman Filter
Types of assimilation algorithms:“smoother” and filter
Day 5 Lecture 1 Module name - Basics on Data Assimilation 21
Why 4D-variational data assimilation?
• provides BLUE (Best Linear Unbiased Estimator). Remark: 3D-var ingested in model does not!
• Potential for “consisteny” within the assimilation intervall (O(1 day))Remark: fast manifold perturbations (=“initialisation problem”) mitigated, but not removed. Inconsistencies at the end of the assimilation windows.
• Allows for extensions to estimate analysis error covariance matrix and temporal correlations (red noise)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 22
How does 4D-var work?
Day 5 Lecture 1 Module name - Basics on Data Assimilation 23
Derivation of 4D-var (1)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 24
Derivation of 4D-var (2)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 25
Derivation of 4D-var (3)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 26
Derivation of 4D-var (4)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 27
Derivation of 4D-var (5)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 28
Derivation of 4D-var (6)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 29
Derivation of 4D-var (7)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 30
CONTRACEConvective Transport of Trace Gases into
the upper Troposphere over
Europe: Budget and Impact of Chemistry
Coord.: H. Huntrieser, DLR
flight path Nov. 14, 2001
Day 5 Lecture 1 Module name - Basics on Data Assimilation 31
CONTRACE Nov. 14, 2001 north (= home) bound
O3
H2O2CO
NO
1. guess assimilation result observations flight height [km]
Day 5 Lecture 1 Module name - Basics on Data Assimilation 32
Collection of assimilation formulavariational data assimilation (3D-var und 4D-var)
Optimal Interpolation (OI)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 33
The “Initialisation” / Filtering Problem (1)• Constraints by physical/chemical equations are
insufficient, as they allow for physically/chemically admissible states, but partly very unlikely to occur: – high amplitude gravity waves (“fast
manifolds”)– pronounced chemical imbalances
• Legacy procedure: introduce penalty term for fast changes (Normal Mode Initialisation, NMI)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 34
The “Initialisation” / Filtering Problem (2)
• or filter function (Digital Filter Initialisation, DFI, here with Lanczos-filter ), x model state:
t=0 model start time
t=Mt=-M t=1
Day 5 Lecture 1 Module name - Basics on Data Assimilation 35
Meteorological remote sensing observation suite (1)
• Atmospheric sounding channels from passive instruments: atmospheric temperature and humidity (Atmospheric sounding channels from the HIRS (High resolution Infrared Sounder) and AMSU (Advanced Microwave Sounding Unit) on board NOAA
• Surface sensing channels from passive instruments "imaging" channels: located in atmospheric "window" regions of the infra-red and microwave spectrum : surface temperature and cloud track information
• Surface sensing channels from active instruments: scatterometeremit microwave radiation for ocean wind retrievals. Some similar-class active instruments such as altimeters and SARS (Synthetic Aperture Radars) provide information on wave height and spectra.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 36
Meteorological remote sensing observation suite (2)
• visible (Lidars) or the microwave (radars) analyse the signal backscattered – molecules, aerosols, water droplets or ice particles. – penetration capability allow the derivation of information on cloud
base, cloud top, wind profiles (Lidars) or cloud and rain profiles
(radars).
• Radio-occulation technique using GPS (Global Positioning System). GPS receivers (e.g. METOP/GRAS) measure the Doppler shift of a GPS signal refracted along the atmospheric limb path.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 37
Observation systems (1)
Type and number of observations used to estimate the atmosphere initial conditions during a typical day. (Buizza, 2000)
dimobservation space= O(106)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 38
Observation systems (2): In-situ observations
Day 5 Lecture 1 Module name - Basics on Data Assimilation 39
Observation systems (3): polar orbiting satellites (e.g. AMSU-A)
Data coverage for the NOAA-15 (red), NOAA-16 (cyan) and NOAA-17 (blue) AMSU-A instruments, for the four 6-hourperiods centred at 00, 06, 12 and 18 UTC 12 November 2002. The plots show the data used for AMSU-A channel 5, which is atemperature-sounding channel in the mid and lower troposphere.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 40
Observation systems (4): geostationary
Data coverage provided by the GOES satellites (cyan and orange) and the METEOSAT satellites (magenta and red) for 00 UTC 10 May 2003. The total number of observations was 266,878.
Source: ECMWF
Day 5 Lecture 1 Module name - Basics on Data Assimilation 41
0
5
10
15
20
25
30
35
40
45
50
55
No. of sources
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009Year
Number of satellite sources used at ECMWF
AEOLUSSMOSTRMMCHAMP/GRACECOSMICMETOPMTSAT radMTSAT windsJASONGOES radMETEOSAT radGMS windsGOES windsMETEOSAT windsAQUATERRAQSCATENVISATERSDMSPNOAA
Satellite data sources in 2007+
Source: ECMWF
Day 5 Lecture 1 Module name - Basics on Data Assimilation 42
Information source: numerical model
nowT799L91(~25 km)
dim O(108)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 43
At which retrieval level to assimilate remote sensing data ?
• Assimilation of retrieved products from space agencies or research institutes (“level 2”)– Pro: most simple for assimilation (assimilation “like in situ observations”)– Con: inexact as xb and B are inconsistent, if retrieval and assimilation
are independent• Locally produced or “1D-Var” or “Averaging Kernel” retrievals
– Pro: xb and B much better known (e.g. fronts featured and consistent with model)
– Con: xb and B used twice with the subsequent assimilation: y and xbcorrelated
• Direct assimilation of radiances (“level 1”)– Pro: retrieval step is essentially incorporated within the main analysis by
finding the model variables that minimize a cost function measuring the departure between the analysed state and both the background andavailable observations.
– Con: Fast linearized radiative transfer scheme and its adjoint is needed.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 44
Example for averaging kernels
Averaging kernels (in K/K) for AIRS (left panel) and HIRS (right panel) instruments.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 45
Background Error Covariance MatrixThe design of the BECM is of utmost importance, as it•weights the model error against the competing observation errors•spreads information from observations to the adjacent area•influences coupled parameters: temperature wind field, chemical constituents (i.e. multivariate correlation)•(serves for preconditioning in the case of variationalassimilation) 2 simple, isotropic and
homogeneouscovariance models:Gaussian
Balgovind
Day 5 Lecture 1 Module name - Basics on Data Assimilation 46
Background Error Covariance Matrix B(2 pragmatic methods to estimate B)
( )( )∑=
−−=K
nj
nji
niij xxxx
KB
1
1Ensemble integration
K= # ensemble members;
i,j grid cells
1. Ensemble approach: (e.g. Evensen, 1994)
2. NMC method: (e.g. Parrish and Derber, 1992)
Difference column vector of model statesof two forecasts at some absolute time ti, one starting 24 hours earlier than the other,thus trying to capture the most sensitive errors.Covariances calculated by outer product.
Day 5 Lecture 1 Module name - Basics on Data Assimilation 47
Construction of the adjoint code
forward model(forward differential equation)
algorithm(solver)
code
backward model(backward differential equation)
adjoint algorithm(adjoint solver)
adjoint code
Day 5 Lecture 1 Module name - Basics on Data Assimilation 48
Constructing the adjoint from forward code
Day 5 Lecture 1 Module name - Basics on Data Assimilation 49
Constructing the adjoint from forward code
Day 5 Lecture 1 Module name - Basics on Data Assimilation 50
Adjoint code verification
Day 5 Lecture 1 Module name - Basics on Data Assimilation 51
Adjoint compilerSome examples• Odyssee• TAMC • Tapenade• ADOL-C and ADOL-F (includes ability for higher
order derivatives)• IMASFor much more comprehensive list see:www.autodiff.org/?module=Tools&submenu=&language=ALL
Also codes from adjoint compilers must be tested!!Algorithm and performance bugs occur!
Day 5 Lecture 1 Module name - Basics on Data Assimilation 52
Adjoint integration Adjoint integration ““backward in timebackward in time””How to make the parameters of resolvents iM(ti-1,ti) available in reverseorder??
Day 5 Lecture 1 Module name - Basics on Data Assimilation 53
Isopleths of the cost function and transformed cost function and minimisation steps
Minimisation by mere gradients, quasi-Newon method L-BFGS(Large dimensional Broyden Fletcher Goldfarb Shanno),and preconditioned (transformed) L-BFGS application
concentration species 1 transformed species 1
conc
entra
tion
spec
ies
2tra
nsfo
rmed
spe
cies
2
Day 5 Lecture 1 Module name - Basics on Data Assimilation 54
Day 5 Lecture 1 Module name - Basics on Data Assimilation 55
Computation of inverse B, square root B, inverse square root Bby (Sca)LAPACK eigenpair decomposition, but better choices available.
Transformed cost functiondefine
transformation
transformed cost function
transformed gradient of the cost function
Pro transformation:minimisation problem is better conditionedContra:strictly positive definite approximation to B required
Day 5 Lecture 1 Module name - Basics on Data Assimilation 56
Kalman filter: basic equationsForecast equations
Analysis equations
Day 5 Lecture 1 Module name - Basics on Data Assimilation 57
Reduced rank Kalman filter (basic idea)Appoximate covariance matrices Pb,a (n x n) by a product of suitably low ranked matrix Sb,a (n x p), and p << n . Same procedure for the system noise matrix Q with T (n x r).
The forecast step remains unchanged.
The forecast error covariance matrix rests on 2 x p model integrations only!
The enlargement of p by r enforces periodic reductions of columns (with lowest ranked eigenvalues)
Day 5 Lecture 1 Module name - Basics on Data Assimilation 58
Reduced rank Kalman filter (calculus)
In practice, all calculations can be performed without actually calculating matrices P!
Positive semidefinitenes is maintained!
Day 5 Lecture 1 Module name - Basics on Data Assimilation 59
Implementing ensemble Kalman filter in practice
Day 5 Lecture 1 Module name - Basics on Data Assimilation 60
εi perturbation vector of potential optimisation parameters: initial values boundary values, emission rates, deposition velocities
C norm inducing pos. def., sym. operator at initial time t0 (Mahalanobis)M tangent linear modelE norm inducing pos. def., sym. operator at optimisation time t1(P projection operator, extinguishing areas or species outside focus)λ Lagrange parameter and generalised eigenvalues
unit constraint (scalar product):maximiseRaleigh quotient:
maximise⇒generalised
EV problem
Some advanced auxiliary calculations (1)Some advanced auxiliary calculations (1)singular value analysis singular value analysis
Day 5 Lecture 1 Module name - Basics on Data Assimilation 61
Assumptions:• Gaussian error distribution assumption sufficiently valid• First guess not too far from “solution” (tangent-linear approximation
must hold)• A priori defined error covariances (background, observations)
Necessary condition for a posteriorivalidation:adjust B and R such that:
ExpectationVariance
Some advanced auxiliary calculations (2)a posteriori validation of data assimilation results
Day 5 Lecture 1 Module name - Basics on Data Assimilation 62
Some advanced auxiliary calculations (3)a posteriori validation of data assimilation results
The partition of costs in terms of all observations and the background at the final analysis
How do the partial costs in the cost function divide?
partial costs of observations (Ip identity matrix in observation space with p observations)
partial costs of background:
Day 5 Lecture 1 Module name - Basics on Data Assimilation 63
The partition of costs in terms of individual classes of information sources zj
Day 5 Lecture 1 Module name - Basics on Data Assimilation 64
Summary• Satellite data still not fully exploited, but
within the scope of variational calculus• 4Dvar is presently regarded as the most
rewarding data assimilation method• Kalman Filtering with suitably reduced
complexity (square root and/or ensemble) feasible, but superiority yet to be proved
Day 5 Lecture 1 Module name - Basics on Data Assimilation 65
Outlook• So far, only systems with Gaussian errors are
treated.• Sophisticted related algorithms (4D-var, KF) are
far from being fully exploited.• However, limits of Gaussian assumptions are
obvious: assimilation for precipitation forecasts and aerosol process poor.
• Further, more general approaches necessary: MCMC, importance resampling, …