The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on. Any opinions, findings and conclusions or recommenda'ons expressed in this publica'on are those of the author(s) and do not necessarily reflect the views of the Na'onal Science Founda'on.
©UCAR 2014
Ensemble Verifica/on: Part II Tim Hoar, NCAR �
Ensemble Verification: definition?�
The atmospheric science literature is rich with ways to quantify the accuracy of a forecast. I am not going to discuss any of these. Whole conferences could be organized around
this topic.��
All too often the poster/presentation states “I pushed the EnKF button and here are the results.” with little to no
regard as to whether or not the assimilation effectively used the information in the observations or whether the ensemble
was useful.��
I am going to focus on metrics that inform about the effectiveness of an ensemble forecast assimilation system.�
�
ICAP tjh 2 of 21
The Big Questions: �
1. Are you looking at the prior (Good!) or the posterior (nearly useless for assimilation assessment – think ‘direct replacement’)?�
2. Do you have an appropriate ensemble?�• Underdispersed …. filter divergence�• Overdispersed … uninformative�• Biased … �
3. Are your metrics skewed by observation rejection?�
4. Where and why are observations being rejected?�ICAP tjh 3 of 21
Introduction to DART�
DART is the Data Assimilation Research Testbed, an �open source, freely available, ensemble data assimilation system that works with many geophysical and dynamical models and many types of observations.��DART is Fortran-based, scales well into the thousands of processors, has tutorials and documentation, and a small support staff.��We also have tools to evaluate the performance of an assimilation system– as opposed to a forecast – I’m going to use those tools to demonstrate what we believe to be important considerations.�
ICAP tjh 4 of 21
DART “home page”: �http://www.image.ucar.edu/DAReS/DART �
Overview article of DART:
Anderson, Jeffrey, T. Hoar, K. Raeder, H. Liu, N. Collins, R. Torn, A. Arellano, 2009: The Data Assimilation Research Testbed: A Community Facility. Bull. Amer. Meteor. Soc., 90, 1283–1296. doi:10.1175/2009BAMS2618.1
Diagnostics�• Was Assimilation Effective?�• Observation-Space Diagnostics�• ... �• Histograms�• State-Space Diagnostics�• …�
ICAP tjh 5 of 21
Schematic for any ensemble system: �
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tk
1. Models advance�
2. Compare the model states to the observation.�
3. Compute increments�
4. Update model states�
1. Models advance�
ICAP tjh 6 of 21
Intro to DART-centric view and tools
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tk
Stat
e-"
"Obs
erva
tion
-�sp
ace"
""s
pace�
ICAP tjh 7 of 21
Intro to DART-centric view and tools
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tk
Prior" " " " "Posterior�
ICAP tjh 8 of 21
Intro to DART-centric view and tools
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tk
Prior�state-space�diagnostics�
Posterior�state-space�diagnostics�
Prior observation-space�diagnostics�
Posterior observation-space�diagnostics* �
*Not really done as pictured, must calculate posterior state and reapply h.�
ICAP tjh 9 of 21
Intro to DART-centric view and tools
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tk
Prior�state-space�diagnostics�
Posterior�state-space�diagnostics�
Can always do these.�
Need some model ‘truth’ to do these.�
Prior observation-space�diagnostics�
Posterior observation-space�diagnostics�
ICAP tjh 10 of 21
Some DART tools for STATE-space diagnostics: �
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tk
Some of these aggregate over a region, �some calculate metrics for each level, �some only make sense if you know the Truth …�
rou/ne purpose
plot_bins.m Plots the rank histograms (usually at a loca'on)
plot_correl.m plot the spa'al correla'on of the ensemble against a single loca'on
plot_ens_'me_series.m Plots the evolu'on of the ensemble (1 loca'on, 1 variable)
plot_total_error.m Plots the RMSE of all variables, all loca'ons* (if Truth is available)
plot_ens_err_spread.m Plots the evolu'on of the ensemble error and spread
ICAP tjh 11 of 21
Some DART tools for OBSERVATION-space diagnostics: �
Anderson: Ensemble Tutorial 14 9/8/06
How an Ensemble Filter Works for Geophysical Data Assimilation
6. When all ensemble members for each state variable are updated,have a new analysis. Integrate to time of next observation...
y
****
h hh
y
tk+2
tkrou/ne purpose
obs_diag.f90 Calculates diagnos'cs from observa'ons, writes out a netCDF file.
plot_evolu'on.m plot the 'me-‐evolu'on of the diagnos'c
plot_profile.m plot a 'me-‐averaged ver'cal profile
plot_rank_histogram.m plot rank histograms (can also be done with ncview)
obs_seq_to_netcdf.f90 creates a netCDF file of the observa'ons
link_obs.m Creates graphics to explore loca'ons/values/QC etc.
Can always do these.�
Prior observation-space�diagnostics�
Posterior observation-space�diagnostics�
ICAP tjh 12 of 21
an example of plot_evolution �
08/01 08/06 08/11 08/160
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
month/day − Aug.01,2005 06:00:00 start
rmse
Northern Hemisphere (20−80)RADIOSONDE_TEMPERATURE @ 500 hPa
forecast: mean=1.1971 analysis: mean=0.98162
forecastanalysis
data file: /Users/thoar/svn/DART/trunk/models/cam/work/obs_diag_output.nc
0
200
400
600
800
1000
1200
1400
1600
1800
2000
# of
obs
: o=
poss
, +=u
sedCan plot:�
RMSE�Bias �Spread�Totalspread�Ens mean �QC value�
prior�
posterior�
# of observations�possible – ‘o’�
# of observations�used – ‘+’�
Simple mean �region �
scal
e fo
r #
of
obse
rvat
ions�
This experiment started from nearly identical conditions and the model dynamics caused it to diverge over time.�
ICAP tjh 13 of 21
evolution of two quantities�
08/01 08/06 08/11 08/160
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
month/day − Aug.01,2005 06:00:00 start
rmse a
nd tota
lspre
ad
Northern Hemisphere (20−80)RADIOSONDE_TEMPERATURE @ 500 hPa
rmse pr=1.1971, po=0.98162 totalspread pr=0.91985, po=0.81559
rmse
totalspread
data file: /Users/thoar/svn/DART/trunk/models/cam/work/obs_diag_output.nc
0
200
400
600
800
1000
1200
1400
1600
1800
2000
# o
f obs : o
=poss, +
=used
Prior and posterior plotted for each timestep … sawtooth.�
Totalspread has a minimum corresponding to observation error.�
RMSE & totalspread similar�
ICAP tjh 14 of 21
compare multiple experiments�
12/26 01/05 01/15 01/25 02/04 02/14 02/24 03/06 03/160
0.1
0.2
0.3
0.4
0.5
0.6
0.7
02−Jan−1998 12:00:01 through 15−Mar−1998 12:00:00
fore
cast
spr
ead
12/26 01/05 01/15 01/25 02/04 02/14 02/24 03/06 03/16
0
7.1
14.3
21.4
28.6
35.7
42.9
50
# of
obs
(o=p
oss,
+=u
sed)
AtlanticXBT_TEMPERATURE @ 20 m
DARTPOP23DARTPOP48
DART also has the ability to calculate the metrics for ‘trusted’ observations, i.e. even when the observation is rejected.�
ICAP tjh 15 of 21
observation-space rank histograms�
plot_rank_histogram�?biased�
ncview can plot rank histograms�right from obs_diag.f90 output.�Can plot several levels on same axis.�
Underdispersed @ 200 hPa �?Overdispersed @ 850 hPa �
This requires the user to save the results of the forward observation operator for all ensemble members during the assimilation. Cannot calculate this ‘after the fact’.�
ICAP tjh 16 of 21
an example of plot_vertical�
Shaded areas correspond to vertical aggregation.�
# of observations�possible – ‘o’�
# of observations�used – ‘+’�
scale for # of obs�
Important not to consider the ‘spin up’ period.�
Priors are solid lines, �posteriors are dashed.�
ICAP tjh 17 of 21
an example of link_obs
ICAP tjh 18 of 21
an example of link_obs
Rotate view angle�
Enable observation selection or ‘brushing’�
e.g. observation �e.g. p
rior
mea
n�
DART QC�
Observa'on processing for forecast metrics:
VariableX: � (analysisT, station, level, copy, ensemble, forecast_lead) | | | | | | | | | | | +- forecast length : 0,3,6,9,12,... | | | | +----------- ensemble member index | | | +------------------ obs value, prior, obs_err | | +------------------------- vertical level index | +--------------------------------- (horizontal) station index +------------------------------------------- analysis time/date
Anything goes: unsorted, unordered observations �
obs_seq_coverage: determines uniform observation network � obs_seq_verify:"
creates a netCDF file with variables ordered to make analysis easy�
ICAP tjh 20 of 21
www.image.ucar.edu/DAReS/DART [email protected]
For more information: �
MITgcm_ocean �
NAAPS�NCOMMAS�
PBL_1d�
POP�
AM2�BGRID �
CAM�
CLM�
COAMPS�
COAMPS_nest �MPAS_OCN �
MPAS_ATM�
NOAH �
PE2LYR �SQG�
WRF�
TIEGCM�
GITM�
CABLE�
WACCM�ROMS� wrfHydro �
ICAP tjh 21 of 21