pdaf - the parallel data assimilation framework: experiences with kalman … · · 2015-11-24pdaf...
TRANSCRIPT
![Page 1: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/1.jpg)
PDAF - The Parallel Data Assimilation Framework:
Experiences with Kalman Filtering
Lars Nerger, Wolfgang Hiller, and Jens Schröter
Alfred Wegener Institute for Polar and Marine Research Bremerhaven, Germany
![Page 2: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/2.jpg)
Overview
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
PDAF in the context of Kalman filters
Parallel performance of PDAF
![Page 3: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/3.jpg)
Data Assimilation
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
Filter ⇔ Smoother
Possible applications:
weather/climate forecastssensitivity studies
Estimate system state (atmosphere, ocean, …) on the basis of a numerical model and measurements by
combining both sources of information.
![Page 4: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/4.jpg)
14-day forecast of ocean surface temperature
Data Assimilation
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 5: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/5.jpg)
Ensemble-based Kalman filters
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
Initialization: Sample initial state and its error estimate by an ensemble of model states.
Forecast: Evolve each ensemble member with the non-linear (stochastic) model.
Analysis: Apply update step of the Kalman filter to ensemble mean or all ensemble states. Error
estimate given by ensemble statistics.
Re-Initialization: Transform state ensemble to exactly represent updated error statistics.
![Page 6: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/6.jpg)
Computational and Practical Issues
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
• Huge amount of memory required(model fields and ensemble matrix)
• Huge requirement of computing time(ensemble integrations)
• Natural parallelism of ensemble integration exists - but needs to be implemented
• Existing models often not prepared for data assimilation
![Page 7: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/7.jpg)
PDAF: Considerations for Implementation
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
Modelinitialization
time integrationpost processing
FilterInitialization
analysisre-initialization
Observationsobs. vector
obs. operatorobs. error
Further considerations• Combination of filter with model with minimal changes
to model code• Control of assimilation program coming from model• Simple switching between different filters and data sets• Complete parallelism in model, filter, and framework
Logical separation of problem
state vector
time
state vector
obs. vector
![Page 8: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/8.jpg)
Extending a Model for Data AssimilationAaaaaaaa
Aaaaaaaa
aaaaaaaaa
Aaaaaaaaaaa
Aaaaaaaa
aaaaaaaaa
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
Filter_MPI_Init
![Page 9: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/9.jpg)
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
• User-supplied routines for
� field transformations between model and filter
� observation-related operations
� filter post-step
• Defined calling interface for
� calls of framework routines
� calls to user-supplied routines
• Interface independent of filter (almost)
PDAF interface structure
![Page 10: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/10.jpg)
2-level Parallelism
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
Model 1
Model 2
Filter
Forecast Filter-Update Forecast
Model 1
Model 2
parallelization variantsdistribute operations
different processes for model and filter update
Filter update with model processes
![Page 11: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/11.jpg)
domain decomposition
parallel filter updatedistribute ensemble matrix
mode decomposition
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 12: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/12.jpg)
Mode decomposition
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 13: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/13.jpg)
Domain decomposition
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 14: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/14.jpg)
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
• Distribute model integrations
• Distribute filter update step
• 3 communicators
� Comm_Model: model tasks
� Comm_Filter: filter processes
� Comm_Couple: communication between model and filter
MPI parallelization
![Page 15: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/15.jpg)
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
• Ensemble Kalman filter (EnKF, Evensen, 1994)
� widely used
� fully nonlinear error forecast
• SEEK filter (Pham et al., 1997)
� explicit low-rank (error-subspace) formulation
� linearized error forecast
• SEIK filter (Pham et al., 1997)
� combination of strengths of EnKF and SEEK
Current KF algorithms in PDAF
![Page 16: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/16.jpg)
3D box experiment
� finite element model FEOM
� 31x31 grid points, 11 layers
� nonlinear problem: interacting baroclinic Rossby waves
� Assimilate sea surface height each 2.5 days over 40 days
(FEOM: Danilov et al., Ocean Modeling, 2004)
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 17: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/17.jpg)
Speedup of PDAF
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 18: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/18.jpg)
Parallel Efficiency of Filter Update
Mode decomposition
Domain decomposition
(ens. size = 10)
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 19: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/19.jpg)
Further Example: FEOM North Atlantic
surface nodes: 16000 3D nodes: 220000z-levels: 23eddy-permitting
![Page 20: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/20.jpg)
Summary
• Parallel Data Assimilation Framework PDAF
� Simplified implementation of assimilation systems
� Flexibility: Different assimilation algorithms anddata configurations within one executable
� Full utilization of parallelism
� High parallel efficiency
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 21: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/21.jpg)
Future directions
� Extensions of PDAF
• more advanced filters (localization, adaptivity)
• smoother algorithms
� Data assimilation applications (oceanography)
• FEOM- stability of North Atlantic circulation
• OPA-Model (with C. Böning, IFM-Geomar, Kiel)- large-scale circulation interannual to decadal
L. Nerger et al., PDAF - The Parallel Data Assimilation Framework
![Page 22: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/22.jpg)
Application: FEOM North Atlantic
L. Nerger et al., Parallel ESKFs
� 3D primitive equation model
� finite-element discretization
Filter Experiments:
� Assimilate synthetic observations of sea surface height ζ
� Covariance matrix estimated from 9-year model trajectory starting from January 1991 initialized from climatology
� Initial state estimate from perpetual 1990 model spin-up
� analysis steps: initial time & after 1 month of model integration
� No model error; forgetting factor 0.8 for both filters
![Page 23: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/23.jpg)
Modeled Sea Surface Height
![Page 24: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/24.jpg)
Estimated Sea Surface Height
![Page 25: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/25.jpg)
Estimated Temperature at -70m
![Page 26: PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman … · · 2015-11-24PDAF - The Parallel Data Assimilation Framework: Experiences with Kalman Filtering Lars](https://reader030.vdocument.in/reader030/viewer/2022020114/5af70c187f8b9a7444903e20/html5/thumbnails/26.jpg)
Comparsion of Computation Times
• Ensemble size 32; 8 concurrent model integrations
Model integrations: 34000s
Filter update:
Difference due to
� inversion of huge matrix in EnKF
� generation of ensemble of observations
10sSEIK
4600sEnKF
TimeFilter