ROMS/TOMS European WorkshopROMS/TOMS European WorkshopAlcala de Henares, Spain, November 7, 2006Alcala de Henares, Spain, November 7, 2006

ean M od

earch C o m

r a i n - F o l l o w

M o d e l i n g

ROMS Framework and AlgorithmsROMS Framework and Algorithms

Andrew M. MooreAndrew M. MooreUCSCUCSC

Emanuele Di LorenzoEmanuele Di LorenzoGeorgia TechGeorgia Tech

Bruce D. CornuelleBruce D. CornuelleSIO, UCSDSIO, UCSD

Arthur J. MillerArthur J. MillerSIO, UCSDSIO, UCSD

Hernan G. ArangoHernan G. ArangoIMCS, RutgersIMCS, Rutgers

John L. WilkinJohn L. WilkinIMCS, RutgersIMCS, Rutgers

Javier Zavala-GarayJavier Zavala-GarayIMCS, RutgersIMCS, Rutgers

The Good… The Bad… The Ugly…

F90 AdjointF90 Adjoint NestingNesting Adjoint MaintenanceAdjoint Maintenance

Data AssimilationData Assimilation Infrequent ReleasesInfrequent Releases Copyright / Open SourceCopyright / Open Source

Adjoint ParallelizationAdjoint Parallelization Grid GenerationGrid Generation ROMS Code DivergenceROMS Code Divergence

Released Version 3.0Released Version 3.0 Open BoundariesOpen Boundaries Compiler BugsCompiler Bugs

WikiROMSWikiROMS DocumentationDocumentation DocumentationDocumentation

Forum ActivityForum Activity Version ControlVersion Control Wetting and DryingWetting and Drying

ROMS BlogROMS Blog Post-processingPost-processing Treatment of RiversTreatment of Rivers

www.myroms.orgwww.myroms.org TutorialsTutorials MonotonicityMonotonicity

Conclusions

ROMS Framework

ROMS Directory Tree

src

Bin

Adjoint

Drivers

External

Include

Modules

Nonlinear

Obsolete

Programs

SeaIce

Utility

Tangent

Representer

Version

Compilers

Lib

makefile

ROMS

Master

SWAN

WRF

ROMS Drivers master.F

#include “cppdefs.h”

#if defined AIR_OCEAN# include "air_ocean.h"#elif defined WAVES_OCEAN# include "waves_ocean.h"#else# include "ocean.h"#endif

ocean_control.F#include “cppdefs.h”

#if defined AD_SENSITIVITY# include "adsen_ocean.h"#elif defined AFT_EIGENMODES# include "afte_ocean.h"#elif defined FT_EIGENMODES# include "fte_ocean.h"#elif defined FORCING_SV# include "fsv_ocean.h"#elif defined OPT_PERTURBATION# include "op_ocean.h"#elif defined OPT_OBSERVATIONS# include "optobs_ocean.h"#elif defined SO_SEMI# include "so_semi_ocean.h"#elif defined S4DVAR# include "s4dvar_ocean.h"#elif defined IS4DVAR# include "is4dvar_ocean.h"#elif defined W4DPSAS# include "w4dpsas_ocean.h"#elif defined W4DVAR# include "w4dvar_ocean.h"#else# if defined TLM_DRIVER# include "tl_ocean.h"# elif defined RPM_DRIVER# include "rp_ocean.h"# elif defined ADM_DRIVER# include "ad_ocean.h"# else# include "nl_ocean.h"# endif#endif

ocean.h#include “cppdefs.h”

PROGRAM ocean

USE ocean_control_mod, ONLY : initialize USE ocean_control_mod, ONLY : run USE ocean_control_mod, ONLY : finalize

#ifdef DISTRIBUTE && defined MPI CALL mpi_init (MyError) CALL mpi_comm_rank (MPI_COMM_WORLD, MyRank, MyError)#endif CALL initialize CALL run CALL finalize

#if defined DISTRIBUTE && defined MPI CALL mpi_finalize (MyError)#endif

END PROGRAM ocean

ROMS Adjoint

• The Adjoint Model (ADM) of ROMS is exact and defined relative to the L2-norm inner-product

• Hand-written using the recipe of Giering and Kaminski (1998)

• Two Tangent Linear Models:

Perturbation Tangent Linear Model (TLM): Generalized Stability Theory (GST) Analyses, Strong and Weak Constraint 4DVar

Finite Amplitude Tangent Linear Model (RPM): Indirect Representers, Weak Constraint 4DVar

• The TLM is derived by linearizing the Nonlinear Model (NLM) around a small perturbation

• The RPM model is derived from the TLM by adding additional terms

Parallelization

• Coarse-grained parallelization: horizontal tiles

• The NLM, TLM, and RPM can be run in either shared-memory (OpenMP) or distributed-memory (MPI)

• The ADM can only be run in distributed-memory (ADM violates shared-memory collision rules)

• Aggregation of variables for MPI communications

CALL ad_mp_exchange2d (ng, iADM, 3, Istr, Iend, Jstr, Jend, &

& LBi, UBi, LBj, UBj, &

& NghostPoints, EWperiodic, NSperiodic, &

& ad_Zt_avg1, ad_DU_avg1, ad_DV_avg1)

Some Uses of Adjoint Models • Data Assimilation: fit model solutions to data by adjusting initial

conditions, boundary conditions and parameters.

• Sensitivity Analysis: study the response of the ocean circulation to variations in all physical attributes of the system

• Eigenmode Analysis: dynamic modes of variability (TLM normal modes, ADM optimal excitations)

• Singular Vectors: stability of the dynamical system (most rapidly growing perturbations)

• Stochastic Optimals: most disruptive patterns of ocean forcing

• Ensemble Prediction: initial condition perturbations along the most unstable directions of the state space

• Adaptive Sampling: design of optimal observational systems

4D Variational Data Assimilation (4DVAR)

• Strong Constraint Conventional (S4DVAR): outer loop, NLM, ADM Incremental (IS4DVAR): inner and outer loops, NLM,

TLM, ADM (Courtier et al., 1994)• Weak Constraint

Indirect Representer Method (W4DVAR): inner and outer loops, NLM, TLM, RPM, ADM (Egbert et al., 1994; Bennett et al, 1997)

Physical Space Statistical Analysis (W4DPSAS): inner and outer loops, NLM, TLM, ADM (Courtier, 1997)

Strong Constraint, Incremental 4DVAR

Let’s introduce a new minimization variable v, such that:

J(vk) = ½(vk)Tvk + ½(Hxk – dk-1)TO-1(Hxk – dk-1)

v J = vk + BT/2HTO-1(Hxk – dk-1) = vk + BT/2x Jo = vk + W -1/2LT/2GS

B = SCS = S(GL1/2W -1/2)(W

-1/2LT/2G)S

xk = B1/2vk + xk-1 – xb

xk = B-1/2(xk + xk-1 – xb)

yielding

The gradient of J in minimization-space, denoted v J, is given by:

The background-error covariance matrix can be factored as:

where S is the background-error standard deviations, C is the background-error correlations which can be factorized as C = C1/2CT/2, G is the normalization matrix which ensures that the diagonal elements of C are equal to unity, L is a 3D self-adjoint filtering operator, and W is the grid cell area or volume.

Model/Background Error Covariance, B

• Use a generalized diffusion squared-root operator (symmetric) as in Weaver et al. (2003):

B = S C S = S (G L1/2

W-1/2

) (W-1/2

LT/2

G)

• The normalization matrix, G, ensure that the diagonal elements of the correlation matrix, C, are equal to unity. They are computed using the exact (expensive) or randomization (cheaper) methods.

• The spatial convolution of the self-adjoint filtering operator, L1/2

, is split in horizontal and vertical components and discretized both explicitly and implicitly.

• The model/background standard deviation matrix, S, is computed from long (monthly, seasonal) simulations.

• The grid cell area or volume matrix, W-1/2

, is assumed to be time invariant.

Strong Constraint, Incremental 4DVAR

Misfit cost function betweenmodel (NLM+TLM) and observations

Cost function gradient

Compute TLM initial conditionsusing first guess conjugategradient step size

Compute change in cost function

Compute TLM initial conditionsUsing refined conjugate gradientstep size

Compute NLM new initial conditions(NLM+TLM)

Compute basic state trajectoryand extract model at observationslocations

Model/Background Error Correlation (C)

Horizontal

Hdecay = 100 km

Vdecay = 100 m

Vertical (implicit)

Model/Background Error Correlation NormalizationCoefficients (G)

SSH Temperature

Bottom Level

EAC EAC

Generalized Stability Theory (GST)

• Dynamics/sensitivity/stability of flow to naturally occurring perturbations

• Dynamics/sensitivity/stability due to error or uncertainties in the forecast system

• Practical applications:

Ensemble prediction

Adaptive observations

Array design ...

Optimal Perturbations

SymmetricProblem

DO n=1, N EV

DO WHI LE (.TRUE.)

NRM2

PROPAGATOR

ESMF

ARPACKDSAUPD

ARPACKDSEUPD AD_MAIN3D

TL_MAIN3D

AD_INI_PERTURB

AD_PACK

Compute ConvergedRitz Eigenvectors

OP_OCEAN

FINALIZE

RUN

PROPAGATOR

TL_UNPACK

INIT IALIZE

A measure of the fastestgrowing of all possibleperturbations over a giventime interval RT(t,0)XR(0,t)u

u

Ensemble Prediction

• Optimal perturbations / singular vectors and stochastic optimals can also be used to generate ensemble forecasts.

• Perturbing the system along the most unstable directions of the state space yields information about the first and second moments of the probability density function (PDF):

ensemble mean

ensemble spread

• Excite with dominant basis vectors

Ensemble Prediction

t

s

HighSpread

U npredic table

timet

sLow

Spread

P redic table

time

For an appropriate forecast skill measure, s

East Australia Current (EAC) ExampleEast Australia Current (EAC) Example

EAC: Incremental 4DVar (IS4DVAR)SSH, SST, XBT Assimilation

Obse

rvati

ons

SSH

SSH

Temperature along XBT line

Temperature along XBT line

Ass

imila

tion

0-d

ays

10-d

ays

SV 2 SV 6 SV 7 SV 8 SV 10

Rit

z E

igenvalu

es

EAC: Optimal Perturbations

+

EAC: Ensemble Prediction

15-days forecast1-day forecast 8-days forecast

Profiling ROMS Kernel: EAC

The NLM is run in the EAC application for 800 time-

steps as standalone driver with the same CPP options

as the TLM, RPM, ADM:

• TLM is 2.5 times more expensive than NLM

• RPM is 2.6 times more expensive than NLM RPM is 3-percent more expensive than TLM

• ADM is 2.8 times more expensive than NLM ADM is 13-percent more expensive than TLM

Linux 2.6.12-15mdksmp #1 SMP Mon Jan 9 23:35:18 MST 2006

x86_64 Dual Core AMD Opteron(tm) Processor 265