rom saf cdop-2 validation report: offline level 3 climate ... · prepared by: hans gleisner rom saf...
TRANSCRIPT
Ref: SAF/ROM/DMI/REP/CLM/001
ROM SAF CDOP-2
Validation Report:
Offline Level 3 Climate Data:
GRM-17, 18, 19, 20, 21
Version 1.2
15 May 2013
Danish Meteorological Institute (DMI) European Centre for Medium-Range Weather Forecasts (ECMWF)
Institut d’Estudis Espacials de Catalunya (IEEC) Met Office (METO)
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
2 of 61
DOCUMENT AUTHOR TABLE
Author(s) Function Date Comment Prepared by: Hans Gleisner ROM SAF Scientist,
Climate Coordinator 15/5 2013
Reviewed by (internal):
Approved by: Kent B. Lauritsen ROM SAF Project Manager 16/5 2013
DOCUMENT CHANGE RECORD
Issue/Revision Date By Description 1.0 12/2 2012 HGL Version for ORR-2.
1.1 15/3 2013 HGL Version for ORR-2 Closeout: addressing RIDs 1-17 and 26-35 from ORR2.
1.2 15/5 2013 HGL Version closing Action 1, points 21 to 26 (raised by A. Steiner) and points 1 to 7 (raised by P. Poli) in Annex 2 of the minutes from the ORR2 Close Out meeting.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
3 of 61
ROM SAF
The Radio Occultation Meteorology Satellite Application Facility (ROM SAF) is a
decentralised processing center under EUMETSAT which is responsible for operational
processing of GRAS radio occultation data from the Metop satellites and radio occultation
(RO) data from other missions. The ROM SAF delivers bending angle, refractivity,
temperature, pressure, and humidity profiles in near-real time and offline for NWP and
climate users. The offline profiles are further processed into climate products consisting of
gridded monthly zonal means of bending angle, refractivity, temperature, humidity, and
geopotential heights together with error descriptions.
The ROM SAF also maintains the Radio Occultation Processing Package (ROPP) which
contains software modules that will aid users wishing to process, quality-control and
assimilate radio occultation data from any radio occultation mission into NWP and other
models.
The ROM SAF Leading Entity is the Danish Meteorological Institute (DMI), with
Cooperating Entities: i) European Centre for Medium-Range Weather Forecasts (ECMWF)
in Reading, United Kingdom, ii) Institut D'Estudis Espacials de Catalunya (IEEC) in
Barcelona, Spain, and iii) Met Office in Exeter, United Kingdom. To get access to our
products or to read more about the ROM SAF please go to: http://www.romsaf.org
Intellectual Property Rights
All intellectual property rights of the ROM SAF products belong to EUMETSAT. The use
of these products is granted to every interested user, free of charge. If you wish to use these
products, EUMETSAT's copyright credit must be shown by displaying the words
“copyright (year) EUMETSAT” on each of the products used.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
4 of 61
List of Contents
1. INTRODUCTION ............................................................................................................................. 5
1.1 PURPOSE OF THE DOCUMENT ......................................................................................................... 5 1.2 APPLICABLE & REFERENCE DOCUMENTS ....................................................................................... 6
1.2.1 Applicable documents ............................................................................................................. 6 1.2.2 Reference documents ............................................................................................................. 6
1.3 ACRONYMS AND ABBREVIATIONS ................................................................................................... 7 1.4 DEFINITIONS .................................................................................................................................... 8
2. BACKGROUND .............................................................................................................................. 9
2.1 LEVEL 1B AND LEVEL 2 PROFILE DATA ........................................................................................... 9 2.2 LEVEL 3 PROCESSING ................................................................................................................... 10 2.3 QUALITY CONTROL ........................................................................................................................ 10 2.4 ECMWF REFERENCE DATA ......................................................................................................... 11
3. VALIDATION ..................................................................................................................................13
3.1 VALIDATION STRATEGY ................................................................................................................. 13 3.2 VALIDATION DATA SET .................................................................................................................. 13 3.3 ZONAL MONTHLY MEANS AND STANDARD DEVIATIONS ................................................................ 17 3.4 ESTIMATES OF A PRIORI INFORMATION ........................................................................................ 28 3.5 ERROR ESTIMATES........................................................................................................................ 34 3.6 COMPARISON TO ECMWF REFERENCE DATA ............................................................................. 45 3.7 VALIDATION AGAINST PRD REQUIREMENTS ................................................................................ 51
3.7.1 PRD requirements ..................................................................................................................... 51 3.7.2 Compliance with PRD requirements ......................................................................................... 52 3.7.3 Service specifications ................................................................................................................ 59
4. OPEN ISSUES ...............................................................................................................................60
4.1 QC PROCEDURES ......................................................................................................................... 60 4.2 SAMPLING ERROR CORRECTION ................................................................................................... 60
5. CONCLUSIONS .............................................................................................................................61
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
5 of 61
1. Introduction
1.1 Purpose of the document
This document describes the validation of the ROM SAF Offline Level 3 climate data
products listed in Table 1. The data products are generated by the ROM SAF processing
system using Level 1b & 2 profile data as input together with ancillary information from
ECMWF analyses and forecasts.
The validation is based on data from the FORMOSAT-3/COSMIC mission (henceforth
referred to as COSMIC). The retrievals of bending angle and refractivity profiles, and the
1D-Var processing, are performed using ROPP software version 6.0 and the ROM SAF
1DV code version 2.5. The Level 3 climate processing was done with version 1.1 of the
ROMCLIM software package.
Section 2 describes the Level 1b and Level 2 profile data used as input to the Level 3
processing. It also includes a brief overview of the Offline Level 3 processing at DMI, the
quality screening process, and a summary of the ECMWF data used as a reference in the
validation. The actual validation, including the data used and the compliance with the
formal requirements, is described in Section 3. A few open issues are pointed out in
Section 4. The main conclusions are stated in Section 5.
An extensive range of plots with a direct bearing on the validation of the climate data can
also be found in the climate monitoring section of the ROM SAF web site
(http://www.romsaf.org). Those plots should be studied in conjunction with the present
report.
Product
identifier
Product name Product
acronym
Mission /
satellite
Dissemination
means
Product
version
GRM-17 Bending angle grid CBACO1 COSMIC Web 1.0
GRM-18 Refractivity grid CRGCO1 COSMIC Web 1.0
GRM-19 Temperature grid CTGCO1 COSMIC Web 1.0
GRM-20 Specific humidity grid CHGCO1 COSMIC Web 1.0
GRM-21 Geopotential height grid CZGCO1 COSMIC Web 1.0
Table 1. Offline Level 3 climate data products.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
6 of 61
1.2 Applicable & reference documents
1.2.1 Applicable documents
The following list contains documents with a direct bearing on the contents of this
document:
[AD.1] CDOP-2 Proposal: Proposal for the Second Continuous Development and
Operations Phase (CDOP-2); Ref: SAF/GRAS/DMI/MGT/CDOP2/001
Version 1.1 of 21 March 2011, approved by the EUMETSAT Council in Ref.
EUM/C/72/11/DOC/10 at its 72nd
meeting on 28-29 June 2011;
[AD.2] CDOP-2 Cooperation Agreement: Agreement between EUMETSAT and DMI
on the Second Continuous Development and Operations Phase (CDOP-2) of
the Radio Occultation Meteorology Satellite Applications Facility (ROM
SAF), approved by the EUMETSAT Council; Ref: EUM/C/72/11/DOC/15 at
its 72nd
meeting on 28-29 June 2011 and signed on 29 June 2011 in
Copenhagen;
[AD.3] ROM SAF Product Requirements Document,
SAF/ROM/DMI/MGT/PRD/001.
[AD.4] ROM SAF Service Specifications, SAF/ROM/DMI/RQ/SESP/001.
1.2.2 Reference documents
The following documents provide supplementary or background information, and could be
helpful in conjunction with this document:
[RD.1] ROM SAF ATBD: 1DVAR algorithms, SAF/ROM/DMI/ALG/1DV/002.
[RD.2] ROM SAF ATBD: Climate algorithms, SAF/ROM/DMI/ALG/CLM/001.
[RD.3] ROM SAF Validation Report: GRM-01 NRT Refractivity Profile (NRP),
SAF/ROM/DMI/RQ/REP/001.
[RD.4] GRAS SAF Validation Report: GRM-02 NRT Temperature profile (NTP), GRM-03
NRT Specific humidity profile (NHP), GRM-04 NRT Pressure profile (NPP), GRM-05
NRT Surface pressure (NSP), SAF/GRAS/DMI/RQ/REP/002.
[RD.5] ROM SAF Validation Report: Offline Level 1b & 2a COSMIC bending-angle &
refractivity profiles, SAF/ROM/DMI/REP/COS/001.
[RD.6] ROM SAF Validation Report: Offline Level 2b COSMIC 1D-Var profiles,
SAF/ROM/DMI/REP/COS/002.
[RD.7] The ROPP User Guide – Part III: Pre-processor module,
SAF/ROM/METO/UG/ROPP/006.
[RD.8] GRAS SAF CT2 Processing Code: Operational Processing of CHAMP and COSMIC
data: Mathematical Methods, Data Filtering and Quality Control,
SAF/GRAS/DMI/ALG/CT2/002.
[RD.9] M.E. Gorbunov and K.B. Lauritsen, Analysis of wave fields by Fourier Integral
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
7 of 61
Operators and its application for radio occultations, Radio Science, 39(4), 4010,
doi:10.1029/2003RS002971.
[RD.10] M. E. Gorbunov, Ionospheric correction and statistical optimization of radio occultation
data, Radio Science, 37(5), 1084, doi:10.1029/2000RS002370, 2002.
[RD.11] Healy, S. B. and Eyre, J. R., Retrieving temperature, water vapor and surface pressure
information from refractive–index profiles derived by radio occultation: A simulation
study, Quart. J. Roy. Meteorol. Soc., 126, 1661–1683, 2000.
[RD.12] Persson and Grazzini. User guide to ECMWF forecast products. Meteorological
Bulletin, M3.2, (available from ECMWF website).
[RD.13] Rieder, M.J., and G. Kirchengast (2001): Error analysis and characterization of
atmospheric profiles retrieved from GNSS occultation data, J. Geophys. Res., vol. 106,
31755.
[RD.14] Foelsche, U., B. Scherllin-Pirscher, F. Ladstädter, A.K. Steiner, and G. Kirchengast,
Refractivity and temperature climate records from multiple radio occultation satellites
consistent within 0.05%, Atmos. Meas. Tech., 4, 2007-2018, 2011.
[RD.15] Schreiner, W., S. Sokolovskiy, D. Hunt, C. Rocken, and Y.-H. Kuo, Analysis of GPS
radio occultation data from the FORMOSAT-3/COSMIC and the Metop/GRAS
missions at CDAAC, Atmos. Meas. Tech., 4, 2255-2272, 2011.
1.3 Acronyms and abbreviations
ATBD Algorithm Theoretical Baseline Document
CDAAC Cosmic Data Analysis and Archival Center
COSMIC Constellation Observing System for Meteorology, Ionosphere and Climate
CDOP-2 Second Continuous Development and Operations Phase (EUMETSAT)
DMI Danish Meteorological Institute; ROM SAF Leading Entity
ECMWF The European Centre for Medium-range Weather Forecasts
EPS EUMETSAT Polar Satellite System
EUMETSAT EUropean organisation for the exploitation of METeorological SATellites
GNSS Global Navigation Satellite System
GPS Global positioning System (US)
GRAS GNSS Receiver for Atmospheric Sounding (EPS/Metop)
GRIB GRIdded Binary (WMO)
IEEC Institut d’Estudis Espacials de Catalunya
L1 GPS carrier frequency, 1575.42 MHz
L2 GPS carrier frequency, 1227.6 MHz
LC L Corrected (through linear combination of L1 and L2)
LEO Low Earth Orbit
Met Office United Kingdom Meteorological Office
Metop Meteorological Operational Polar satellite (EUMETSAT)
MSL Mean Sea Level
netCDF Network Common Data Format
NRT Near Real Time
NWP Numerical Weather Prediction
PRD Product Requirements Document (ROM SAF)
RO Radio Occultation
ROM SAF Radio Occultation Meteorology SAF (former GRAS SAF)
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
8 of 61
ROPP Radio Occultation Processing Package (ROM SAF)
SAF Satellite Application Facility (EUMETSAT)
WMO World Meteorological Organization
WWW World Weather Watch (WMO programme)
1.4 Definitions
RO data products from the GRAS instrument onboard Metop and RO data from other data
providers are grouped in levels and are either NRT or Offline products:
Data levels:
Level 0: Raw sounding, tracking and ancillary data, and other GNSS data before clock
correction and reconstruction;
Level 1a: Reconstructed full resolution excess phases, SNR’s, amplitudes, orbit
information, I, Q, and NCO values, and navigation bits;
Level 1b: Bending angles and impact parameters, Earth location, metadata and quality
information;
Level 2: Refractivity profiles (level 2a), and pressure, temperature, and specific
humidity profiles (level 2b and 2c), Earth location, metadata, and quality information;
Level 3: Gridded level 1 and 2 offline profile products in the form of, e.g., monthly
and seasonal zonal means, metadata, and quality information;
Product types:
NRT product: data product delivered less than 3 hours after measurement;
Offline product: data product delivered less than 30 days after measurement (the
timeliness for some offline level 3 products may be up to 6 months);
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
9 of 61
2. Background
2.1 Level 1b and Level 2 profile data
The two GPS radio frequencies (L1 and L2) received by an RO instrument onboard a LEO
satellite are characterised by their amplitude and phase values. The bending angle profiles
are obtained using the positions and velocities of the GPS and LEO satellites [RD.7]. The
raw bending angle profiles are first subject to a correction in order to eliminate the effect of
the ionosphere on the signals (LC). In the case of single ray propagation the phase contains
all the necessary information in order to derive the bending angle whereas in the case of
multiple ray propagation (multipath), caused by strong vertical gradients in the atmosphere,
both the amplitude and the phase are needed to obtain a bending angle profile free of
multipath artifacts. In Offline processing both the phase and the amplitude are used to solve
for the multipath propagation that occurs frequently in the troposphere [RD.8,9].
The observed, raw bending angles undergo ionospheric correction and statistical
optimization, whereby the observed data are smoothed and merged with an MSIS-90
profile mapped to bending angle space [RD.7,10]. A refractivity profile is then obtained
through an inversion using the Abel transform [RD.7]. To retrieve the ROM SAF products
temperature, pressure and humidity (Level 2b), ancillary data are needed. These data are
obtained from ECMWF forecasts, appropriate to the time and location of the occultation. In
combination with the observed refractivity, the ECMWF background is used in a 1D-Var
algorithm to simultaneously estimate the temperature, humidity and pressure profiles,
together with surface pressure [RD.7,11]. The solution is constrained by the assumption
that the atmosphere is in hydrostatic equilibrium. Note that unique humidity profiles cannot
be obtained from radio occultation measurements without using some source of ancillary
information on temperature, something referred to as the temperature-humidity ambiguity.
The starting point for the ROM SAF Level 3 processing is vertical profiles of bending
angle () as a function of impact parameter, microwave refractivity (N) as a function of
mean-sea level altitude (H; geometric height above the geoid), and temperature (T),
pressure (p), and specific humidity (q) as functions of mean-sea level altitude, H. The
geopotential height (Z) as a function of pressure or pressure height (Hp) is an alternative
formulation of the pressure profile [RD.2]
The bending angles and the refractivities are provided on relatively dense vertical grids
reaching up to around 100 km. i.e. above the region where the RO technique provides
useful information on the neutral atmosphere. The temperature, pressure, geopotential
height, and specific-humidity profiles are given on a standard set of ECMWF model levels
ranging from the surface up to around 80 km.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
10 of 61
2.2 Level 3 processing
The climate data are generated from the RO atmospheric profile data through rather
straight-forward binning and averaging [RD.2]. A set of equal-angle latitudinal bands, or
grid boxes, are defined and all valid observations that fall within a latitude band and
calendar month undergo a weighted averaging to form an area-weighted zonal mean for
that latitude and month. The error in the mean is estimated as a combination of the per-
profile observational errors and errors due to under-sampling of the atmosphere.
The RO climate data products are generated by the following steps:
listing of all occultations that were observed within a calendar month
quality control followed by removal of profiles that are deemed “bad”
vertical interpolation of profiles onto the regular climate height grids
weighted averaging into monthly latitude bins
estimation of the errors (observational and sampling) in the zonal monthly means
estimation of the a priori information in the zonal monthly means
formatting of the RO climate data and meta-data into image and data files
The generation of zonal monthly mean climate data may be followed by further averaging
into seasonal and annual means, and into regional, hemispheric, and global means. Further
details of the Level 3 processing are found in [RD.2].
2.3 Quality control
As described in Section 2.2, the Level 3 climate data are generated from Level 2 profile
data. The quality of the climate data depends on the quality of the input profile data
[RD.5,6], and the quality control (QC) procedure is a central part of the Level 3 processing.
It is a stepwise procedure that starts with a basic check to ensure that:
the bending angle profile reaches above 60 km and below 30 km impact altitude
the bending angle profile contains more than 100 valid data points
the impact altitudes for the bending-angle profile form a monotonic series
all bending angles are within the range [-1,100] mrad
the refractivity profile reaches above 60 km and below 30 km altitude
the refractivity profile contains more than 100 valid data points
the altitudes of the refractivity profile form a monotonic series
all refractivities are within the range [0,500] N-units
The number of data points required for the bending angle and refractivity profiles depends
on the sampling density of the profiles.
Next, the noise properties of the L2 signal and the degree of fit of the raw LC bending
angle to the background bending angle is checked. The L2 quality score quantifies the
degradation of the L2 signal through the RMS difference of the L1 and L2 impact
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
11 of 61
parameter series obtained from a radio-holographic analysis [RD.7,8]. The SO quality score
quantifies the relative error (expressed as a percentage) of the neutral bending angle,
estimated by the degree of best fit to a background bending angle profile [RD.10]. The
settings currently used for these two quality scores are:
the L2 quality score must be less than 30.0
the SO quality score must be less than 1000.0
It is the L2 quality score that dominates the QC screening, whereas the SO quality score
currently is set to a value that only gives it a minor role. The third QC step removes data
that do not seem to belong to the distribution, through a comparison with ECMWF
reference data:
the refractivity profiles must deviate from ECMWF forecasts by less than 10%
between 10 km and 35 km
The above QC screening applies to all data. For the data derived from a 1D-Var procedure
(pressure, temperature, humidity) there is a fourth QC step which provides a check on the
1D-Var solution:
the 1D-Var retrieval algorithm must converge within 25 iterations
the normalized penalty function 2J/Nobs must be smaller than 5.0
2.4 ECMWF reference data
ECMWF short-term forecasts are used both as an a priori in the 1D-Var retrieval of Level
2b profile data [RD.1] and as a reference in the validation of the Level 3 climate data
products [RD.12]. The ECMWF fields are retrieved in the form of GRIB files from the
standard dissemination stream at a 1.0˚x1.0˚ resolution [RD.12]. The use of one and the
same resolution in the 1D-Var processing and in the reference data set allows us to
investigate which information that the observations contributed to the monthly means.
Each occultation has a nominal latitude, longitude, and time associated with it. To retrieve
co-located ECMWF profiles, we first select the ECMWF forecast at the nearest 6-hourly
time step (i.e. at UTC 0, 6, 12, or 18) and then interpolate bi-linearly from the ECMWF
grid to the nominal latitude and longitude of the occultation. Since new ECMWF forecast
streams are started at 0 UTC and 12 UTC the forecasts used in the nominal situation are as
follows:
0:00 UTC - 3:00 UTC: 12-hour forecast for 0 UTC (based on the analysis at 12 UTC)
3:00 UTC - 9:00 UTC: 6-hour forecast for 6 UTC (based on the analysis at 0 UTC)
9:00 UTC - 15:00 UTC: 12-hour forecast for 12 UTC (based on the analysis at 0 UTC)
15:00 UTC - 21:00 UTC: 6-hour forecast for 18 UTC (based on the analysis at 12 UTC)
21:00 UTC - 24:00 UTC: 12-hour forecast for 24 UTC (based on the analysis at 12 UTC)
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
12 of 61
For each observed profile we thus obtain a corresponding ECMWF model profile.
Following a forward-modeling step, this model profile can be used as a reference in the
validation.
ECMWF data are also used in the Level 3 processing to estimate sampling errors. These
data are taken from analysis fields on a coarser 2.5˚x2.5˚ degrees horizontal resolution. The
latter grid has been interpolated from an ECMWF spectral model truncated to T63, which
has a horizontal resolution comparable to the observation resolution of an RO instrument.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
13 of 61
3. Validation
3.1 Validation strategy
The purpose of the validation is to demonstrate that the ROM SAF Level 3 data products
meet the standards expected from an operational EUMETSAT data product, and in
particular that they meet the formal requirements as stated in the ROM SAF PRD [AD.2].
The adopted validation procedure aims to demonstrate that the deviations of the COSMIC
monthly mean data from the ECMWF reference data (described in Section 2.4) are
consistent with the required accuracies stated in the PRD (summarized in Table 2).
The formal validation covers two full years: 2010 and 2011. The present report shows
detailed examples of zonal plots for two representative months (January and July 2011). In
addition to this, an extensive range of plots – zonal, monthly, and time series plots for
climate variables as well as diagnostic variables – can be found on the associated ROM
SAF climate monitoring web site (http://www.romsaf.org/climate_monitoring) which
should be studied in conjunction with this report. Many of the plots on the web page have a
direct bearing on the validation of the climate data.
The PRD specifies three sets of requirements for each of the five monthly mean data
products (“threshold”, “target”, and “optimal” accuracies). The requirements are further
separated into two height ranges (0–15 km and 15–40 km), except for the humidity
requirements which only have a single height range (0–12 km). In the lower height range,
the requirements for bending angle and refractivity vary linearly with height. The full set of
PRD requirements applicable to the Level 3 climate data are listed in Table 2, Section 3.7.
Even though the requirements are separated into two height ranges, the actual comparison
between the observed deviations and the formal requirements are performed separately in
three height layers (0-8 km, 8-15 km, and 15-40 km). The formal compliance with the
requirements is summarized in Section 3.7, where the 70% percentile of the absolute
deviation of observed monthly means from the ECMWF reference data are compared with
the PRD requirements. This choice of formal test is roughly in line with what is currently
used in the Level 2 validation, in which one standard deviations are used for comparison
with the PRD requirements (for a normal distribution 68% of the data fall within +/- 1
standard deviation). The 80% and 90% percentiles are also shown for information. The
formal part of the validation is performed for all months in the 2-year validation period,
and in addition to this the climate monitoring web site provides updated validation
information for the full length of the climate data set.
3.2 Validation data set
The validation period consists of the two years 2010-2011. Postprocessed excess-phase
data (version 2010.2640) were obtained from the UCAR’s COSMIC Analysis and
Archiving Center (CDAAC). After conversion from CDAAC’s atmPhs file format to the
ROPP file format, these data were further processed from excess phase (Level 1a) to
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
14 of 61
bending angle (Level 1b), refractivity (Level 2a), pressure, temperature and humidity
(Level 2b) profile data using ROM SAF processing software [RD.1,7]. The QC procedures
described in Section 2.3 are applied to the profile data, and the data not rejected by the QC
are then used as input to the Level 3 processing in which the gridded climate data are
generated [RD.2]. The ECMWF short-term forecasts used as a reference in the validation
are taken from GRIB fields at a 1.0˚x1.0˚ horizontal resolution (see Section 2.4).
The number of occultations available for climate data generation varies with time (Fig. 1).
After 2009 this variation is considerable for the COSMIC mission, partially related to
power problems onboard the COSMIC satellites, which have exceeded their nominal
lifetime of 5 years. The COSMIC-1 and 6 satellites in particular caused a loss of data
during the time period, e.g. the apparent seasonal decrease in the number of occultations
passing QC3 can be traced to these two satellites. The QC screening further reduces the
number of data that can be used to generate climate data.
The data numbers are relatively uniform with longitude, and also with local time (Fig. 2;
upper panels). The latter is a consequence of the fact that measurements are made from
several satellites in different orbits, and that the orbits drift in local time. On the other
hand, the data numbers vary substantially with latitude, predominantly a consequence of
the GNSS and LEO satellite orbits in combination with the limb-sounding observing
geometry of the RO instrument (Fig. 3; middle panels). The RO antenna characteristics
also act to modulate these distributions.
Figure 1. Number of occultations available for climate data generation, after the consecutive QC steps described in Section 2.3. QC0 is a fundamental sanity check of bending-angle and refractivity profiles, QC2 is a check based on the L2 and SO quality scores, QC3 consists of systematic removal of outliers through comparison with ECMWF, and QC4 is a check on the 1D-Var solution. The large variation in the number of occultations is partially explained by problems with power onboard the COSMIC satellites, which have exceeded their nominal life time of 5 years.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
15 of 61
Figure 2. Upper panels: distribution of occultations over longitude (left) and over local time (right) for July 2010. On a monthly time scale these distributions are relatively uniform for the COSMIC
mission. Middle panels: distribution of occultations over latitude, for equal-angle latitude bands (left) and for equal-area latitude bands (right). The latitudinal distributions are predominantly governed by the GNSS and LEO satellite orbits, and by the limb-sounding observing geometry of
the RO instrument, and vary relatively little with time. Lower panels: scatter of occultations in azimuth/longitude (left) and in azimuth/latitude (right). These characteristic patterns are a consequence of satellite orbits and observing geometries.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
16 of 61
Figure 3. Number of samples per grid box (upper panel) and relative number of samples (upper panel) for refractivity in July 2010. Data numbers are the same for bending angle, and slightly lower for the 1D-Var quantities (temperature, humidity, geopotential height). The distribution over latitude described by Fig. 2 is clearly visible, as well as a drop-off of data numbers with altitude.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
17 of 61
3.3 Zonal monthly means and standard deviations
For each latitude-height grid box, we compute the monthly means and the corresponding
standard deviations [RD.2]. These are shown in Figures 4a-e and 5a-e, respectively. The
standard deviation gives an indication of the degree of atmospheric variability.
The monthly mean fields in Figures 4a-e show the overall structure expected for winter and
summer months.
The bending angle and refractivity standard deviations show the effects of enhanced
atmospheric variability at the boundary to the stratospheric winter-hemisphere polar
regions. Enhanced variability is also evident in the sub-tropical troposphere. The
temperature and humidity standard deviations indicate that the refractivity variations are
dominated by temperature variability in the high-latitude stratosphere, and by humidity
variability in the low-latitude troposphere.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
18 of 61
Figure 4a: Bending angle zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
19 of 61
Figure 4b: Refractivity zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
20 of 61
Figure 4c: Temperature zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
21 of 61
Figure 4d: Humidity zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
22 of 61
Figure 4e: Geopotential height zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
23 of 61
Figure 5a: Bending angle zonal monthly standard deviations for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
24 of 61
Figure 5b: Refractivity zonal monthly standard deviations for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
25 of 61
Figure 5c: Temperature zonal monthly standard deviations for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
26 of 61
Figure 5d: Humidity zonal monthly standard deviations for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
27 of 61
Figure 5e: Geopotential height zonal monthly standard deviations for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
28 of 61
3.4 Estimates of a priori information
The grid-box means described in Section 3.3 are not a result of observational data alone,
but also depend on a priori data – background atmospheric data taken from a model. There
are two sources of a priori information: (a) background bending angle profiles that are used
to smooth the observed bending angles and extend them to infinity before inverting
bending angles to refractivity, and (b) the background atmospheric states that are used to
resolve the temperature-humidity ambiguity through a 1D-Var procedure.
As described in the ATBD [RD.2], we define the parameter
2
obs
2
bg
2
obs
2
obs
2
bg
2
bg
SO 1
W (1)
where obs and bg are estimates of the errors in the observed and background bending
angles. The parameter WSO provides a measure of the observational information in the
optimized bending angle profile, which is inverted to refractivity through an Abel
transform. As a consequence of the error characteristics, WSO is a monotonic function of
altitude and goes from 0 (no a priori information) at low altitudes to 1 (no observational
information) at high altitudes. The corresponding zonally averaged quantity (with the ratio
normalized to percent) is shown in Fig. 6a.
Through the Abel inversion, the a priori fraction in refractivity is propagated further down
to lower altitudes. We estimate the resulting a priori fraction through
A
WAW SO
ref 100 (2)
where is a bending angle profile and A[ ] denotes the Abel transform. The factor 100
normalizes the ratio to percent. The refractivity a priori fraction is estimated individually
for all profiles, and the corresponding zonally averaged quantity is shown in Fig. 6b.
Following the notation in the ATBD [RD.2], we let O and B denote the observational and
background error covariance matrixes assumed in the 1D-Var retrieval. The solution error
covariance matrix S is then given by
HOHBS-1-1-1 T (3)
in the linear limit, i.e. in the limit where the forward model H(x) can be represented by its
Jacobian H. The second term on the right hand side is the observational error covariances
mapped to the background state space. Following Rieder and Kirchengast [RD.13] we let
the error standard deviations – i.e. the square root of the diagonal elements of the error
covariance matrix – quantify the relative importance of the a priori information in the
retrieved temperature and humidity:
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
29 of 61
bgT,
solT,
T 100
W (4)
bgq,
solq,
q 100
W (5)
where index ‘sol’ denotes solution and index ‘bg’ denotes background. The factor 100
normalizes the ratio to percent. The corresponding zonally averaged quantities are shown
in Figs. 6c-d.
The impacts by the background (MSIS) bending angle used in the statistical optimization
procedure are described by Fig. 6a. In general, the impacts are larger at high latitudes. In a
cold, dry atmosphere, the effects of the statistical optimization reach deeper into the
atmosphere. Note that the bending angle monthly means are generated from the raw, non-
optimized bending angle profiles. Hence, the MSIS a priori only enters the refractivity and
the following products.
The impacts by the background used in the 1D-Var retrieval are quantified in Figs. 6c-d.
The temperatures have a broad height range from around 8 km up to 35 km where the
observational information is considerable. For humidity, the observational information is
concentrated to heights below 10 km, and is greatest at low- and mid-latitudes.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
30 of 61
Figure 6a: Estimates of the a priori fraction in the statistically optimized bending angle (only used to generate the refractivity) for January and July 2011. A value of zero means there is only observational data in the corresponding monthly means.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
31 of 61
Figure 6b: Estimates of the a priori fraction in the refractivity for January and July 2011. A value of zero means there is only observational data in the corresponding monthly means.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
32 of 61
Figure 6c: Estimates of the a priori fraction in the 1D-Var temperatures for January and July 2011. A value of zero means there is only observational data in the corresponding monthly means.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
33 of 61
Figure 6d: Estimates of the a priori fraction in the 1D-Var humidity for January and July 2011. A value of zero means there is only observational data in the corresponding monthly means.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
34 of 61
3.5 Error estimates
The observed grid-box mean described in Section 3.4 is an estimate of the true grid-box
mean. The difference between the observed mean and the true mean is referred to as the
error of the climate variable. This error is assumed to be caused by two effects. First, each
measurement has a random observational error or measurement error associated with it.
This error can only be described in terms of a statistical uncertainty. Secondly, the finite
number of measurements is not able to fully account for all variability within the grid box
and time interval, resulting in a sampling error. Unlike the observational errors, it is
possible to estimate the actually realized sampling errors affecting the monthly means.
As a part of the operational climate processing, we specify the assumed observational
uncertainty and provide an estimate of the sampling error associated with the monthly
mean. Detailed descriptions of the assumptions and estimates are found in [RD.2].
The observational errors of the monthly means are based on the observational errors per
profile, scaled with the square root of the weighted number of observations in a grid box.
The observational errors for bending angle profiles are 6% at the surface, decreasing
linearly to 0.6% at the tropopause (here simply taken to be 12 km). Above the tropopause,
the errors are 0.6% or 2 rad, whichever is greatest. These numbers are roughly in line with
the results from several studies of the RMS differences between collocated profiles
observed by different COSMIC satellites soon after launch [e.g., RD.15]. The
corresponding errors in the refractivity profiles are from 2% at the surface to 0.2% at the
tropopause, and above that the greatest of 0.2% and 0.02 N-units. The observational errors
for temperature, pressure, and humidity profiles are the formal errors obtained as a part of
the 1D-Var solution [RD.1]. Hence, the errors vary between profiles, but normally we find
temperature errors in the troposphere around 0.8-1.5 K, and relative errors in the specific
humidity amounting to 20-50%, or even more [RD.6].
The sampling errors are estimated from sampling an ECMWF analysis field at the nominal
time and locations of the observations. A weighted average is generated from the sampled
ECMWF data and the difference to the true mean, obtained from averaging the full grid,
provides an estimate of the sampling error [RD.2].
As described in the ATBD, the observational and sampling errors are independent and can
be combined into a total error according to
2
samp
2
obsclim (6)
Figure 7a-e show the assumed observational uncertainties in the zonal monthly means for
the five climate variables. Figures 8a-e show the corresponding estimates of the sampling
errors. Both show a considerable variation with latitude and height. The distribution of the
observational uncertainties for bending angle and refractivity (Figures 7a and 7b) is a
simple consequence of the assumed per-profile errors and the distribution of the number of
samples per grid box, whereas the sampling errors are related to the variability as indicated
by the grid box standard deviations. The sampling errors can be roughly an order of
magnitude larger than the observational uncertainties, but also show a stronger variability.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
35 of 61
Figure 7a: Estimates of the bending-angle observational uncertainty in the zonal monthly means for January and July 2011. The number of samples per grid box varies from around 200 to more than 1200, and the observational uncertainty of the mean scales accordingly.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
36 of 61
Figure 7b: Estimates of the refractivity observational uncertainty in the zonal monthly means for January and July 2011. The number of samples per grid box varies from around 200 to more than 1200, and the observational uncertainty of the mean scales accordingly.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
37 of 61
Figure 7c: Estimates of the temperature observational uncertainty in the zonal monthly means for January and July 2011. The number of samples per grid box varies from around 200 to more than 1200, and the observational uncertainty of the mean scales accordingly.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
38 of 61
Figure 7d: Estimates of the specific-humidity observational uncertainty in the zonal monthly means for January and July 2011. The number of samples per grid box varies from around 200 to more than 1200, and the observational uncertainty of the mean scales accordingly.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
39 of 61
Figure 7e: Estimates of the geopotential-height observational error in the zonal monthly means for January and July 2011. The number of samples per grid box varies from around 200 to more than 1200, and the observational uncertainty of the mean scales accordingly.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
40 of 61
Figure 8a: Estimates of the bending-angle sampling error in the zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
41 of 61
Figure 8b: Estimates of the refractivity sampling error in the zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
42 of 61
Figure 8c: Estimates of the temperature sampling error in the zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
43 of 61
Figure 8d: Estimates of the specific-humidity sampling error in the zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
44 of 61
Figure 8e: Estimates of the geopotential height sampling error in the zonal monthly means for January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
45 of 61
3.6 Comparison to ECMWF reference data
In this section, we demonstrate the difference between the observed zonal monthly means
and monthly means generated from co-located ECMWF forecasts. In Section 3.7, we then
compare statistics of the absolute values of these differences with the requirements
specified in the PRD [AD.2].
There are a few issues to consider when comparing monthly-mean data generated from
observed RO data with ECMWF data. First, ECMWF analyses have assimilated COSMIC
RO data. We should therefore try to avoid comparing COSMIC with ECMWF analyses.
Secondly, ECMWF forecast data have been used as an a priori in the 1D-Var processing of
the Level 2 profile data used as input to the Level 3 processing.
These two factors complicate the choice of reference for the comparison. We have chosen
to use the ECMWF forecast data described in Section 2.4. For bending angle and
refractivity, the comparison should therefore be unaffected by impacts on the ECMWF data
by the RO data itself. Impacts on RO data by ECMWF data will, however, affect the
comparisons of the temperature, humidity, and geopotential height, particularly at altitudes
and/or latitudes where the a priori dominates the 1D-Var solution (see Section 2.4).
Figs. 9a-e show the difference between the observed monthly means and the monthly
means generated from co-located ECMWF forecast, for two months in 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
46 of 61
Figure 9a: Deviations of COSMIC monthly means from the corresponding collocated ECMWF short-term forecast field during January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
47 of 61
Figure 9b: Deviations of COSMIC monthly means from the corresponding collocated ECMWF short-term forecast field during January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
48 of 61
Figure 9c: Deviations of COSMIC monthly means from the corresponding collocated ECMWF short-term forecast field during January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
49 of 61
Figure 9d: Deviations of COSMIC monthly means from the corresponding collocated ECMWF short-term forecast field during January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
50 of 61
Figure 9e: Deviations of COSMIC monthly means from the corresponding collocated ECMWF short-term forecast field during January and July 2011.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
51 of 61
3.7 Validation against PRD requirements
3.7.1 PRD requirements
The formal requirements for the ROM SAF data products are stated in the PRD [AD.2].
The particular requirements applicable to the Level 3 climate data products are here
reiterated in Table 2. There are three sets of requirements for each climate data product
(threshold, target, and optimal). The requirements are further separated into two height
layers (0-15 km and 15-40 km), except for the humidity requirements which only covers 0-
12 km. In the lower height range, the requirements for bending angle and refractivity vary
linearly with height.
As indicated in Table 2 below, the accuracy requirements have been color coded: threshold
is yellow, target is light green, and optimal is dark green. For each climate variable and
month, and within specified latitudinal bands and height layers (see Section 3.7.2), we
define one observed quality figure to compare with the requirements. If this reaches
threshold, but not target, the variable is indicated by yellow color for that month, latitude
band and height layer. If the quality figure reaches target, but not optimal accuracy, it is
indicated by light green color. Dark green is shown if the quality figure reaches optimal
accuracy. Red color means that the quality figure does not reach the threshold value
specified by the PRD.
PRD
REQUIREMENTS
Threshold
0–15 km
(0-12 km)*
Target
0–15 km
(0-12 km)*
Optimal
0–15 km
(0-12 km)*
Threshold
15–40 km
Target
15–40 km
Optimal
15–40 km
CBA: bending angle 1.0 – 4.0 % 0.4 – 2.0 % 0.2 – 1.0 % 2.5 rad 1.0 %
1.0 rad 0.4 %
0.5 rad 0.2 %
CRG: refractivity 0.6 – 2.0 % 0.3 – 1.0 % 0.1 – 0.5 % 0.6 % 0.3 % 0.1 %
CTG: temperature* 1 K 0.2 K 0.1 K 2 K 0.4 K 0.2 K
CHG: spec. humidity 0.25 g/kg
10 % 0.05 g/kg
2 % 0.03 g/kg
1 % N/A N/A N/A
CZG: geopot. height 50 gpm 12 gpm 5 gpm 50 gpm 12 gpm 5 gpm
Table 2. Formal requirements in terms of the threshold, target, and optimal accuracies in two height layers; 0–15 km and 15–40 km (for humidity: 0–12 km), according to the PRD [AD.2]. In the cases where both absolute and relative numbers are given, the requirement is given by the largest of these two. The colour codes are used in Tables 3 to 7 to refer to the accuracy levels met by the actual data quality measures.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
52 of 61
3.7.2 Compliance with PRD requirements
The zonal field is separated into three latitudinal zones: tropics (30˚S–30˚N), mid-latitudes
(30˚N–60˚N and 30˚S–60˚S), and polar regions (60˚N–90˚N and 60˚S–90˚S). Each of these
zones is further separated into three height layers (0-8 km, 8-15 km, and 15-40 km). Since
the climate grid is 200 meters by 5 degrees latitude there are several hundred monthly
values of the absolute deviation from ECMWF, di, in each of the nine latitude-height
regions. Each value, di, is compared with a certain PRD requirement, Ri, valid for that
particular height (and, possibly, latitude). If more than 70% of the values
i
ii
R
dq (7)
within a latitude-height region are smaller than one (i.e., di is smaller than Ri), the observed
data comply with that particular requirement. Differently stated, the 70% percentile of the
distribution of qi’s should be smaller than one,
170 q (8)
for compliance. In the upper panels of Tables 3 to 7, the percentile q70 is compared to the
PRD requirements for every month in the two year validation period 2010-2011. As
described in Section 3.7.1, the color codes refer to the most strict PRD accuracy level
reached.
Hence, the formal compliance with the PRD requirements is based on the 70% percentiles
of the absolute deviation from ECMWF. This quantity is also used in the formal
comparison with the Service Specification [AD.4], and is roughly in line with the current
validation of Level 2 data which use one standard deviation values to compare with PRD
requirements (68% of the samples fall within +/- 1 standard deviation for a normal
distribution).
For bending angle climatologies, with only a few exceptions the 70% percentile values of
the absolute deviations from ECMWF are within target. In the 8-15 km layer they all reach
optimal values. Most of the 80% percentile values also reach target, but a larger fraction
only reach threshold values, particularly in the lower tropical troposphere. This is even
more pronounced for the 90% percentile values which often are outside of threshold in the
lower tropical troposphere, but otherwise reach either target or threshold, and in the 8-15
km layer even optimal values.
The refractivity climatologies show a better compliance to the PRD requirements in the
troposphere than the bending angles. They are never outside of threshold in the 0-8 km
layer, not even for the 90% percentile values. On the other hand, in the polar stratosphere a
fraction of the 80% percentile values are outside of threshold. On these occasions, the 70%
percentile values do not reach target but are still within threshold. The largest deviations of
the refractivity climatologies from the PRD requirements appear to occur high in the
stratosphere during southern hemisphere winter.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
53 of 61
For the 1D-Var quantities (temperature, humidity, and geopotential height) the comparison
of the absolute deviations from ECMWF is only meaningful where the observational
information is significant (see Section 3.4). For the 1D-Var temperatures that would
primarily be the atmospheric regions above 8 km, while for the 1D-Var humidity the
observational information is significant below 8 km and at low- and mid-latitudes. We find
that the 70% percentile values always reach target, except for humidity which show a few
values that only reach threshold. The 80% and 90% percentile values exhibit more cases
that only reach threshold, but they are never outside of threshold.
From these comparisons, we conclude that the observed COSMIC climate data comply
with the Level 3 requirements specified in the PRD [AD.2].
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
54 of 61
Table 3a-c. Comparison of the 70%, 80%, and 90% percentile values of the absolute deviation from ECMWF, with the formal PRD requirements (Table 2). Bending angle during 2010-2011.
70%
80%
90%
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
55 of 61
Table 4a-c. Comparison of the 70%, 80%, and 90% percentile values of the absolute deviation from ECMWF, with the formal PRD requirements (Table 2). Refractivity during 2010-2011.
70%
80%
90%
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
56 of 61
Table 5a-c. Comparison of the 70%, 80%, and 90% percentile values of the absolute deviation from ECMWF, with the formal PRD requirements (Table 2). 1D-Var temperature during 2010-2011.
70%
80%
90%
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
57 of 61
Table 6a-c. Comparison of the 70%, 80%, and 90% percentile values of the absolute deviation from ECMWF, with the formal PRD requirements (Table 2). 1D-Var specific humidity during 2010-2011.
70%
80%
90%
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
58 of 61
Table 7a-c. Comparison of the 70%, 80%, and 90% percentile values of the absolute deviation from ECMWF, with the formal PRD requirements. 1D-Var geopotential height during 2010-2011.
70%
80%
90%
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
59 of 61
3.7.3 Service specifications
The Service Specifications [AD.4] describes the commitments by the ROM SAF related to
the services and products provided to the users. These commitments include a set of
operational accuracy targets that should be met by the Level 3 climate data products, and
which should be regularly monitored and documented as a part of normal operations. Even
though these targets should be consistent with the PRD [AD.3], they are not necessarily
identical to the PRD requirements.
The accuracy targets defined in the service specifications for the Level 3 climate data
products GRM-17…21 are listed in Table 8 below. They are defined similarly to the target
accuracies in the PRD requirements, described in Section 3.7.1.
We monitor the compliance to the service specification by comparing the 70% percentiles
of the absolute deviation from the ECMWF reference data with the accuracies stated in the
Service Specifications Document [AD.4]. Hence, the procedure is similar to the validation
described in Section 3.7.2. The outcome of the regular monitoring against the service
specifications is provided on the ROM SAF web page.
SERVICE
SPECIFICATIONS
Accuracy
0–15 km
(0-12 km)*
Accuracy
15–40 km
CBA: bending angle 0.4 – 2.0 % 1.0 rad
0.4 %
CRG: refractivity 0.3 – 1.0 % 0.3 %
CTG: temperature 0.2 K 0.4 K
CHG: spec. humidity* 0.05 g/kg
2 % N/A
CZG: geopot. height 12 gpm 12 gpm
Table 8. Service specifications according to the Service Specifications Document [AD.4]. The accuracies are stated separately in two height layers; 0–15 km and 15–40 km (for humidity: 0–12 km). In the cases where both absolute and relative accuracies are stated, the accuracy is given by the largest of these two. For bending angle and refractivity, the specified accuracies vary linearly with height in the lower layer.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
60 of 61
4. Open issues
A few open issues remain that will be addressed in future updates of the Level 3 climate
data generation procedures. The issues of highest priority concern the QC procedures and
the correction of the climate data for sampling errors.
4.1 QC procedures
During the validation period about 1000-2000 occultations became available for processing
every day. Of these, about 5% are rejected by the first QC step (the basic sanity check; see
Section 2.3). Another 15% of the original occultations are rejected by the L2 quality-score
criterium in the second QC step. The SO quality-score criterium only affects a few
occultations. In the third QC step, roughly 1% of the original profiles are rejected by the
comparison of observed refractivity with ECMWF. The QC screening of the 1D-Var
solution finally discards a few of the temperature and pressure profiles, typically 0.1-0.3%
of the original data.
The number of profiles rejected is relatively large due to intentionally conservative settings
of the QC criteria – we want to be sure that no erroneous data significantly affect the
climate statistics. These settings will be revised in future updates of the processing
software.
4.2 Sampling error correction
Studies have shown that the errors in the climatological means can be reduced by
subtraction of the estimated sampling errors [RD.14]. This is not done in the present
version of the climate data products. The data provided to the users are the original
climatologies, and not the sampling error subtracted climatologies. However, the estimated
errors are provided as a part of the climate data product, and users may consider to apply a
sampling error correction to the monthly means.
Ref: SAF/ROM/DMI/REP/CLM/001 Issue: 1.2 Date: 15 May 2013
ROM SAF CDOP-2 Validation Report:
GRM-17, 18, 19, 20, 21
61 of 61
5. Conclusions
During the 2-year validation period 2010-2011, on average about 1600 COSMIC
occultations per day were available in the form of post-processed excess phase data. After
QC screening, around 80% of these occultations were available for processing into Level 3
climate data at the ROM SAF.
The results in this report, and the related plots on the ROM SAF climate monitoring web
page, show that the ROM SAF Level 3 climate data are of sufficient quality for use in
science studies and in climate monitoring applications.
The deviations of the observed climatologies from the ECMWF reference data (based on
collocated ECMWF short-term forecast profiles) are consistent with the requirements
specified by the PRD. The 70% percentile values of the deviations are well within the
limits, while a small fraction of the 80% and 90% percentile values for bending angles and
refractivity are outside the limits stated in the PRD.
A few open issues are pointed out, but none of them are judged to be critical for the use of
the climate data.