Download - Data Assimilation
![Page 1: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/1.jpg)
Data Assimilation
Tristan Quaife, Philip Lewis
![Page 2: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/2.jpg)
What is Data Assimilation?
A working definition:
The set techniques the combine data with some underlying process model to provide optimal estimates of the true state and/or parameters of that model.
![Page 3: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/3.jpg)
What is Data Assimilation?
It is not just model inversion.
But could be seen as a process constraint on inversion (e.g. a temporal constraint)
![Page 4: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/4.jpg)
e.g. Use EO data to constrain estimates of terrestrial C fluxes
Terrestrial EO data: no direct constraint on C fluxes
Combine with model
![Page 5: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/5.jpg)
Data Assimilation is Bayesian
• Bayes’ theorem:
P(A|B) =P(B|A) P(A)
P(B)
![Page 6: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/6.jpg)
What does DA aim to do?
Use all available information about The underlying model The observations The observation operator
Including estimates of uncertainty and the current state of the system
To provide a best estimate of the true state of the system with quantified uncertainty
![Page 7: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/7.jpg)
Kalman Filter DA: MODIS LAI product
Data assimilation into DALEC ecological model
![Page 8: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/8.jpg)
Lower-level product DA
Ensure consistency between model and observations
Assimilate low-level products (surface reflectance)
Uncertainty better quantified
Need to build observation operator relating model state (e.g. LAI) to reflectance
Example of Oregon (MODIS DA)
Quaife et al. 2008, RSE
![Page 9: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/9.jpg)
Modelled vs. observed reflectance
Red NIRNote BRF shape in red: can’t simulate with 1-D canopy (GORT here)
![Page 10: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/10.jpg)
NEP results
No assimilation
Assimilating MODIS
(red/NIR)
DALEC model calibrated from flux measurements at tower site 1
![Page 11: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/11.jpg)
Integrated flux predictions
Flux (gC.m-2)
Assimilated data
3yr totalStandardDeviation
NEP
No assimilation 240.2 212.2
MODIS B1 & B2 373.0 151.3
Williams et al. (2005)
406.0 27.8
GPP
No assimilation 1646.4 834.5
MODIS B1 & B2 2620.3 96.8
Williams et al. (2005)
2170.3 18.1
Flux (gC.m-2)
Assimilated data
TotalStandardDeviation
NEP
Assimilation exc. snow
373.0 151.3
Assimilation inc. snow
404.8 129.6
Williams et al. (2005)
406.0 27.8
GPP
Assimilation exc. snow
2620.3 96.8
Assimilation inc. snow
2525.6 42.7
Williams et al. (2005)
2170.3 18.1
![Page 12: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/12.jpg)
Mean NEP for 2000-2002
15 65
gC/m2/year
4.5 km
Flux Tower
Spatial average = 50.9
Std. dev. = 9.7
(gC/m2/year)
![Page 13: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/13.jpg)
NEP – Site2 (intermediate) parameters, with/without DA
Model running at Site 2, Oregon
Site 1 model EO-calibrated at site 2NEP observations from Site 2
Shows ability to spatialise
![Page 14: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/14.jpg)
Data assimilation
Low-level DA can be effective
‘easier’ data uncertainties
Can be applied to multiple observation types
Requires Observation operator(s)
RT models
Requires other uncertainties
Ecosystem Model
Driver (climate)
Observation operator
![Page 15: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/15.jpg)
Specific issues in land EOLDAS
No spatial transfer of information Require full spatial coverage Atmosphere dealt with by an instantaneous retrieval
(i.e. no transport model) All state vector members influence observations
We are not interested in other variables!
![Page 16: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/16.jpg)
Sequential Smoothers Variational
Nominal classification of DA
![Page 17: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/17.jpg)
Kalman Filter Variants - EKF
Ensemble Kalman Filter Variants – Unscented EnKF
Particle filters Lots of different types true MCMC technique
Sequential methods
![Page 18: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/18.jpg)
• Propagation step:
x = Mx-
P = MP-MT + Q• Analysis step:
x* = x + K( y – Hx )
K = PHT( HPHT+R )-1
The Kalman filter
State vectorModel
Covariancematrix
Stochastic forcingKalman
gainObservation
vectorObservation covariance
matrix
Observation operator
![Page 19: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/19.jpg)
The Kalman Filter
• Linear process model
• Linear observation operator
• Assumes normally distributed errors
![Page 20: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/20.jpg)
• Propagation step:
x = m(x-)
P = M'P-M'T + Q• Analysis step:
x* = x + K( y – h(x) )
K = PH'T( H'PH'T+R )-1
The Extended Kalman filter
Jacobian matrix
Jacobian matrix
![Page 21: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/21.jpg)
The Extended Kalman Filter
• Linear process model
• Linear observation operator
• Assumes normally distributed errors
• Problem with divergence
![Page 22: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/22.jpg)
• Propagation step:
X = m(X-) + Q
no explicit error propagation• Analysis step:
X*= X + K( D – HX )
K = PHT( HPHT+R )-1
The Ensemble Kalman filter
State vector ensemble
Perturbed observations
![Page 23: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/23.jpg)
The Ensemble Kalman Filter
• P estimated from X
• Non linear observations using augmentation:
Xa = h(X) X
![Page 24: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/24.jpg)
The Ensemble Kalman Filter
No assimilation
Assimilating MODIS surface
reflectance bands 1 and 2
![Page 25: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/25.jpg)
The Ensemble Kalman Filter
• Avoids use of Jacobian matrices
• Assumes normally distributed errors
– Some degree of relaxation of this assumption
• Augmentation assumes local linearisation
![Page 26: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/26.jpg)
Particle Filters
• Propagation step:
X = m(X-) + Q
• Analysis step:
e.g. Metropolis-Hastings
![Page 27: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/27.jpg)
Particle Filters
![Page 28: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/28.jpg)
Particle Filters
No available observations
![Page 29: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/29.jpg)
Particle Filters
• Fully Bayesian
– No underlying assumptions about distributions
• Theoretically the most appealing choice of sequential technique, but…
• Our analysis show little difference with EnKF
• Potentially requires larger ensemble
– But comparing 1:1 is faster than EnKF
![Page 30: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/30.jpg)
Sequential techniques
• General considerations:
– Designed for real time systems
– Only consider historical observations
– Only assimilates observations in single time step
– Can lead to artificial high frequency components
![Page 31: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/31.jpg)
• Extension of sequential techniques
• All observations effect every time step
• Analogous to weighting on observations
– [ smoothing-convolution / regularisation ]
• Difficult to apply in rapid change areas
Smoothers
![Page 32: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/32.jpg)
Smoothers - regularisation
x = (HTR-1H + λ2BTB)-1HTR-1y
B is the required constraint. It imposes:
Bf = 0and the scalar λ is a weighting on the constraint.
Constraint matrix
Lagrange multiplier
![Page 33: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/33.jpg)
Regularisation
![Page 34: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/34.jpg)
Regularisation
Quaife and Lewis (2010) Temporal constraints on linear
BRDF model parameters. IEEE TGRS, in press.
![Page 35: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/35.jpg)
Regularisation
• Lots of literature on the selection of λ
– Cross validation etc
• Permits insight into the form of Q
![Page 36: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/36.jpg)
Variational techniques
• Expressed as a cost function
• Uses numerical minimisation
• Gradient descent requires differential
• Traditionally used for initial conditions
– But parameters may also be adjusted
![Page 37: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/37.jpg)
3DVAR
J(x) = ( x-x- ) P-1 ( x-x- )T +
( y-h(x) ) R-1 ( y-h(x) )T
Background
Observations
![Page 38: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/38.jpg)
3DVAR
• No temporal propagation of state vector
– OK for zero order approximations
– Unable to deal with phenology
![Page 39: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/39.jpg)
4DVAR
J(x) = ( x-x- ) P-1 ( x-x- )T +
( y-h(xi) ) R-1 ( y-h(xi) )TΣi
Time varying state vector
![Page 40: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/40.jpg)
Variational techniques
• Parameters constant over time window
• Non smooth transitions
• Assumes normal error distribution
• Size of time window?
• For zero-order case 3DVAR = 4DVAR
– 4DVAR for use with phenology model
• Absence of Q - propagation of P?
![Page 41: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/41.jpg)
Building an EOLDAS
• Lewis et al. (RSE submitted)
• Sentinel-2
![Page 42: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/42.jpg)
EOLDAS
![Page 43: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/43.jpg)
Assimilation
Assume model
Uncertainty known
![Page 44: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/44.jpg)
EOLDAS
Base level noise
![Page 45: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/45.jpg)
Cross validation
![Page 46: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/46.jpg)
Cross validation
![Page 47: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/47.jpg)
EOLDAS
Cross validation
![Page 48: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/48.jpg)
Double noise
![Page 49: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/49.jpg)
EOLDAS
Double noise
![Page 50: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/50.jpg)
Conclusions - technique
• DA is optimal way to combine observations and model
• Range of options available for DA
• Sequential
• Smoothers
• Variational
• Require understanding of relative uncertainties of model and observations
• Require way of linking observations and model state
• Observation operator (e.g. RT)
![Page 51: Data Assimilation](https://reader031.vdocument.in/reader031/viewer/2022020219/56815a95550346895dc8136e/html5/thumbnails/51.jpg)
References
• P. Lewis et al. (2010 submitted) RSE, An EOLDAS
• T. Quaife, P. Lewis, M. DE Kauwe, M. Williams, B. Law, M. Disney, P. Bowyer (2008), Assimilating Canopy Reflectance data into an Ecosystem Model with an Ensemble Kalman Filter, Remote Sensing of Environment, 112(4),1347-1364.
• T. Quaife and P. Lewis (2010) Temporal constraints on linear BRF model parameters IEEE Transactions on Geoscience and Remote Sensing doi: 10.1109/TGRS.2009.2038901
• http://www.ecmwf.int/newsevents/training/rcourse_notes/DATA_ASSIMILATION/ASSIM_CONCEPTS/Assim_concepts11.html
• http://www.cs.unc.edu/~welch/kalman/