EOLDAS_ngThe new generation

Prof. P. Lewis, Dr. Jose Gomez-Dans, UCL/NCEO

Task 3: EO-LDAS baselineWP3.1: EO-LDAS v1 consolidation

o Autumn 2013/workshopo Publicly available software & documentation

WP3.2: Integration of enhanced componentso Dec/2013o Integration of other modelling efforts in variational

DA

What is EO-LDAS?• generic variational DA system, weak or strong constraint

• different observation operators (RT models)

• different process models, including regularisation

• tutorial documentation -- how to do DA with the system

• Demonstrated so far with optical data

Lewis et al (2012), Rem Sens Env

Lewis et al (2012), ESA SP

The concept

Observations(RT codes)

Dynamic Model Prior

Define a cost function J as a sum of however many components, but typically:

• Each component calculates the cost and the partial derivatives of the cost function• Use gradient descent minimisation

o Need partial derivatives for Observation Operator & Dynamic model• Uncertainty is calculated using the Hessian (matrix of 2nd order derivatives) @ minimum

J, J'

What have we learned with EO-LDAS?• v1 design controlled by configuration files

o minimise programming requirementso generic structure probably soundo teach users new metalanguage to describe problem

• Too much flexibility leads to inefficiencieso Difficult to add extra functionalityo Difficult to maintain

EO-LDAS v21. Assume user knows Python (e.g. I/O, plotting)2. Streamline system by reducing flexibility3. Simple components - readily extended4. Decouple solver from operators

a. Essentially, list of J, J' EO-LDAS should be a Python library of state

definitions and generic operators, requiring the user to interface to other:• ObsOps,• Dynamic Models,• Data,• Plotting, etc

Interactive documentationIPython notebooks to provide interactive documentation on library brary

http://nbviewer.ipython.org/urls/github.com/jgomezdans/eoldas_ng/raw/master/notebooks/ASimpleSmoother.ipynb

Faster models through emulation• First steps in using Gaussian Processes (GP)

Provides a fast version of the code• ... but also of its partial derivatives!

1. User generates training set with RT model2. GP is trained to the above3. This is used in EO-LDAS

• Much more flexible: o if the model can be emulated, any model can go ino no need to waste time on adjointso very fasto simplifies library structure very much

Emulator testingUse IPython notebooks to provide the user with interactive documentation on how to do emulationhttp://nbviewer.ipython.org/urls/raw.github.com/profLewis/emulate_test/master/emulate_test.ipynb

Spares

Multi-resolution processing

GotchasObservation Operator gotchas• Typical RT codes are science grade software:

o Limited support from the authorso Inefficiento Difficult to follow & modify

• Few RT models have adjoints readily available• If the RT model is modified, the adjoint needs to be re-created

Dynamic model gotchas• All of the above• Ended up implemented a linear model: if you can express your model as Ax, then you can slot

it in• Most dynamic models are highly non-linear, and linearisation requires automatic differentiation• So ended up using Tikhonov-type regularisation