regional production quarterly report on the daily analyses
TRANSCRIPT
Regional productionQuarterly report on the daily analyses and forecasts activities, and verification of the LOTOS-EUROS performances
December 2015 – January 2016 – February 2016
Copernicus Atmosphere Monitoring Service
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances |
This document has been produced in the context of the Copernicus Atmosphere Monitoring
Service (CAMS). The activities leading to these results have been contracted by the
European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the
European Union (Delegation Agreement signed on 11/11/2014). All information in this
document is provided "as is" and no guarantee or warranty is given that the information is
fit for any particular purpose. The user thereof uses the information at its sole risk and
liability. For the avoidance of all doubts, the European Commission and the European Centre
for Medium-Range Weather Forecasts has no liability in respect of this document, which is
merely representing the authors view.
Copernicus Atmosphere Monitoring Service
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances |
Quarterly report on the dailyanalyses and forecasts activities, and verification ofthe LOTOS-EUROS performances
December 2015 – January 2016 – February 2016
KNMI (J. Douros, H. Eskes)
TNO (A. Segers, R. Timmermans)
METEO-FRANCE (M. Pithon, M. Plu, J. Parmentier,J. Arteta, S. Guidotti, N. Assar)
Date: 04/05/2016
REF.:CAMS50_2015SC1_D50.3.2.LOTOS-EUROS-2016Q1_201605
CAMS50_2015SC1_D50.3.4.LOTOS-EUROS-2016Q1_201605CAMS50_2015SC1_D50.5.1.LOTOS-EUROS-2016Q1_201605
Copernicus Atmosphere Monitoring Service
Contents:
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances |
1. Executive Summary.........................................................................................42. The LOTOS-EUROS model (KNMI-TNO)...............................................................5Product portfolio..................................................................................................5Availability statistics.............................................................................................5
Use for observations for data assimilation...............................................................73. Verification report..........................................................................................10Verification of NRT forecasts................................................................................10Verification of NRT analyses................................................................................15
Copernicus Atmosphere Monitoring Service
1. Executive Summary
The Copernicus Atmosphere Monitoring Service (CAMS, www.copernicus-atmosphere.eu) is establishing the core global and regional atmospheric environmental
service delivered as a component of Europe's Copernicus programme. The regionalforecasting service provides daily 4-days forecasts of the main air quality species andanalyses of the day before, from 7 state-of-the-art atmospheric chemistry models andfrom the median ensemble calculated from the 7 model forecasts. The regional service
also provides posteriori reanalyses using the latest validated observation datasetavailable for assimilation.
This report covers the deliverables related to Near Real Time Production (NRT) for
LOTOS-EUROS: D50.3.2-2016Q1, D50.3.4-2016Q1, D50.5.1-2016Q1, for the quarterDecember 2015 – January 2016 – February 2016. Verification is done against in-situsurface observations; they are described in the report D50.1.2-2016Q1, that will bedelivered shortly. The verification of analyses is done against non assimilated
observations.
During this quarter, the performance of LOTOS-EUROS forecasts and analyses wassignificantly affected by a number of bugs associated with the reintroduction of the use
of C-IFS boundary conditions, as well as changes of format in the observational dataprovided for assimilation. The bugs were fixed and the updated model version has beenused in the LOTOS-EUROS production stream since the 1st of March 2016, i.e. just afterthe end of this quarter. It is worth mentioning however, that despite those problems,
LOTOS-EUROS was able to meet the RMSE reference targets for daily maxima andmeans in all occasions.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 4
Copernicus Atmosphere Monitoring Service
2. The LOTOS-EUROS model (KNMI-TNO)
Product portfolio
Name Forecast Analysis
Description Forecast at surface, 50m,
250m, 500m, 1000m, 2000m,
3000m, 5000m above ground
Analysis at the surface
Available for users at 4:00 UTC 10:00 UTC for the day before
Species O3, NO2, CO, SO2, PM2.5, PM10,
NO, NH3, NMVOC, PANs,
Birch pollen at surface during
season
O3, NO2, CO, SO2, PM2.5, PM10,
NO
Time span 0-96h, hourly 0-24h for the day before,
hourly
Availability statistics
The statistics below describe the ratio of days for which the LOTOS-EUROS modeloutputs were available on time to be included in the ENSEMBLE fields (analyses and
forecasts) that are computed at METEO-FRANCE. They are based on the followingschedule for the provision at METEO-FRANCE of:
- forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from49h to 96h)
- analyses data: before 11:00 UTC
These schedules have been set to meet the IT requirements for ENSEMBLE products(no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses).
Indicators:
Availability_model_Forecast
Quarterly basis
D0: 99%
D1: 99%
D2: 99%
D3: 99%
Availability _model_Analysis
Quarterly basis
D: 88%
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 5
Copernicus Atmosphere Monitoring Service
During this quarter, the following issues have been encountered by the LOTOS-EUROSproduction system:
Date Problem description
(origin, effects)
Impact
on production
From 03/12/2015 to
11/12/2015
Delay in the provision of
analysis results
Analysis results arrived too
late to be taken into account
as a member in ENSEMBLE
15/12/2015 Delay in the provision of
analysis results
Analysis results arrived too
late to be taken into account
as a member in ENSEMBLE
19/12/2015 Analysis results missing Analysis results not
participating to the ENSEMBLE
calculation
29/12/2015 New year issue (2016 input
not found) that led to a delay
in the forecast provision
Forecast results late arrival
(too late for participating to
ENSEMBLE)
The three first entries in the table above concern the same problem, which was due to
the establishment of 11:00 UTC as the delivery deadline for the daily analyses, whichwas rather early based on the runtimes of the LOTOS-EUROS analysis runs. Theproblem was diagnosed and corrective action was taken on the 21st of December 2015.
At the moment, eight (8) ensemble members are used in the Ensemble Kalman filter ofthe LOTOS-EUROS assimilation suite, instead of the “standard” twelve (12). We plan torevert to twelve ensemble members, as soon as a number of different optimizationswill be implemented in the model code, which are expected to considerably speed up
model runtimes.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 6
Copernicus Atmosphere Monitoring Service
Use for observations for data assimilation
Day
Use of observation for LOTOS-EUROS
December
O3 NO2 NO SO2 CO PM10 PM2.5
1 8,542
2 8,492
3 8,814
4 8,993
5 8,679
6 8,701
7 8,606
8 8,586
9 9,087
10 9,042
11 8,990
12 9,075
13 8,432
14 3,417
15 9,191
16 8,414
17 3,520
18 8,593
19 3,857
20 8,242
21 3,732
22 8,910
23 9,003
24 8,560
25 8,360
26 689
27 0
28 701
29 929
30 1,033
31 1,044
average 6,524
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 7
Copernicus Atmosphere Monitoring Service
Day
Use of observation for LOTOS-EUROS
January
O3 NO2 NO SO2 CO PM10 PM2.5
1 1,527
2 1,843
3 286
4 3,831
5 6,078
6 8,527
7 1,347
8 7,845
9 1,328
10 4
11 2,623
12 6,341
13 9,108
14 8,907
15 3,559
16 2,546
17 2,458
18 9,416
19 9,150
20 9,358
21 9,327
22 9,308
23 9,171
24 8,955
25 9,300
26 4,249
27 9,005
28 9,059
29 9,208
30 8,705
31 8,903
average 6,170
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 8
Copernicus Atmosphere Monitoring Service
Day
Use of observation for LOTOS-EUROS
February
O3 NO2 NO SO2 CO PM10 PM2.5
1 9,149
2 2,533
3 9,214
4 8,989
5 9,027
6 9,018
7 8,892
8 8,907
9 8,765
10 8,416
11 8,730
12 7,936
13 7,733
14 7,977
15 8,348
16 8,418
17 7,867
18 4,528
19 0
20 8,062
21 8,096
22 8,141
23 7,040
24 8,156
25 8,132
26 5,980
27 212
28 210
29 0
average 6,844
Comments: days with zero or limited number of assimilated observations are due tolimited availability of observations.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 9
Copernicus Atmosphere Monitoring Service
3. Verification report
This verification report covers the period December 2015 – January 2016 – February2016. The LOTOS-EUROS skill scores are successively presented for four pollutants:ozone, NO2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to96h (hourly values), allowing to evaluate the entire diurnal cycle and the evolution of
performance from day 0 to day 3. The forecasts and the analyses cover a largeEuropean domain (25°W-45°E, 30°N-70°N). The statistical scores that are reported arethe root-mean-square error, the modified mean bias and the correlation. The surfaceobservations that are acquired by METEO-FRANCE and used for verification are
described in D50.1.2-2016Q1, that will be delivered shortly.
Verification of NRT forecasts
The following figures present, for each pollutant (ozone, NO2, PM10, PM2.5):
- in the upper-left panel, the root-mean square error of daily maximum (for ozone and
NO2) or of daily mean (PM10) for the first-day forecasts with regards to surfaceobservations, for every quarter since DJF2014/2015, a target reference value isindicated as an orange line,
- in the upper-right panel, the root-mean square error of pollutant concentration
forecasts with regards to surface observations as a function of forecast term,
- in the lower-left panel, the modified mean bias of pollutant concentration forecastswith regards to surface observations as a function of forecast term,
- in the lower-right panel, the correlation of pollutant concentration forecasts with
regards to surface observations as a function of forecast term.
The graphics show the performance of LOTOS-EUROS (black curves) and of theENSEMBLE (blue curves).
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 10
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS forecasts: ozone skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
The performance of LOTOS-EUROS as regards ozone during this quarter was slightlyworse than that of the same quarter of the previous year, but manages nevertheless to
meet the target reference value for the RMSE of the daily maxima. The modelunderestimates more strongly during the early afternoon hours, which however is thecase for the ENSEMBLE mean as well. Correlation is considerably lower that of theENSEMBLE mean and decreases with forecast time.
To a certain extent, the performance of LOTOS-EUROS during this quarter was affectedby a bug in the C-IFS BCs code implementation which was responsible for latitudes ofthe C-IFS domain to be read in the wrong order. This has led to inconsistencies in the
influxes from the lateral boundaries. The bug was corrected in a code update whichtook place on the 1st of March 2016, therefore after the end of the quarter underdiscussion.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 11
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS forecasts: NO2 skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
The behaviour of LOTOS-EUROS as regards NO2 during this quarter was very similarwith the same quarter of the previous hear and manages to meet the target reference
value for the RMSE of the daily maxima. Bias (MMB) is consistently negative and veryclose to the values of the ENSEMBLE mean, except during the early morning hourswhen the underestimation is relatively stronger. Correlation is slightly better and withless fluctuation compared to last years values, although about 0.1 points lower than
those of the ENSEMBLE. The primarily local character of NO2 concentrations couldprovide a reasonable clue as to why the impact of the bug in the C-IFS BCsimplementation mentioned before, does not appear significant in this case.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 12
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS forecasts: PM10 skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
The LOTOS-EUROS forecast performance for PM10 deteriorated considerably during thisquarter. This is exhibited by a moderately positive bias (instead of the negative bias of
the ENSEMBLE and LOTOS-EUROS during the same quarter last year) and much higherRMSE, again compared to the ENSEMBLE and last year's results. The reason behindthis deterioration is the reintroduction of the use of C-IFS boundary conditions on the3rd November 2015. Further to the aforementioned bug that was responsible for
reversed lateral boundary conditions, the use of C-IFS allowed for unrealistically largeamounts of sea salt to enter the western boundary of the computational domain, whichdepending on the prevailing meteorological conditions lead to overestimatedconcentrations over mainland Europe for both PM10 and PM2.5. As a temporary measure
to ameliorate this effect, the use of C-IFS BCs for sea salt was discontinued on the 1st
of March 2016 (i.e. after the end of the quarter under discussion), at the same timewhen the aforementioned bug was fixed.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 13
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS forecasts: PM2.5 skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
The LOTOS-EUROS forecast performance for PM2.5 deteriorated during this quarter in amanner that resembles the performance of PM10. The reasons for this are explained in
the comments on PM10 performance.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 14
Copernicus Atmosphere Monitoring Service
Verification of NRT analyses
The following figures present, for each pollutant (ozone, NO2, PM10):
- in the upper-left panel, the root-mean square error of daily maximum (for ozone andNO2) or of daily mean (PM10) for the analyses (solid line) and for the first-day forecasts(dashed line) with regards to surface observations, for every quarter since
DJF2014/2015, a target reference value is indicated as an orange line,
- in the upper-right panel, the root-mean square error of pollutant concentration of theanalyses (solid line) and of the first-day forecasts (dashed line), with regards tosurface observations as a function of forecast term,
- in the lower-left panel, the modified mean bias of pollutant concentration forecasts ofthe analyses (solid line) and of the first-day forecasts (dashed line), with regards tosurface observations as a function of forecast term,
- in the lower-right panel, the correlation of pollutant concentration of the analyses
(solid line) and of the first-day forecasts (dashed line), with regards to surfaceobservations as a function of forecast term.
The graphics show the performances of LOTOS-EUROS (black curves) and of theENSEMBLE (blue curves). The superposition of the analysis scores (solid lines) and of
the forecast scores (dashed lines) computed over the same observation dataset ishelpful to assess the added value of data assimilation.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 15
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS analyses: ozone skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
The LOTOS-EUROS ozone analyses scores show a rather negative bias and RMSE thatis clearly larger than the that of the ENSEMBLE forecasts. However, RMSE for the daily
maxima is marginally below the reference target. Correlation is lower than that of theENSEMBLE. The reason for the relatively poor performance of the LOTOS-EUROSanalyses lies to a bug that was discovered in January (2016) and was responsible formisidentifying species other than ozone in the observational data (in most cases PM2.5),
as ozone. The effect of the bug was most pronounced during summertime, whenobserved ozone concentrations are generally high and at the same time PM2.5 wasgenerally lower than in the wintertime. The bug was corrected in the operationalproduction stream on the 2nd of February.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 16
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS analyses: NO2 skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
LOTOS-EUROS analyses did not use ground based NO2 observations for the periodsunder consideration.
The LOTOS-EUROS FC RMSE of daily maximum value meets the target value for thisquarter, while the diurnal profiles generally convey the same picture as detailed in theprevious set of graphs for NO2.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 17
Copernicus Atmosphere Monitoring Service
LOTOS-EUROS analyses: PM10 skill scores against data from representativesites, period December 2015 - January 2016 - February 2016
LOTOS-EUROS analyses did not use ground based PM10 observations for the periodsunder consideration.
The LOTOS-EUROS FC RMSE of daily maximum value meets the target value but theeffects of CIFS Bcs are evident, and have obviously led to a deterioration ofperformance since the previous quarter.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 18
Copernicus Atmosphere Monitoring Service
Analysis of LOTOS-EUROS performances for the quarter
The performance of the LOTOS-EUROS model this quarter was hurdled by twoimportant bugs in the model code used to pre-process input data. In conjunction withan intrinsic C-IFS overprediction of sea salt concentrations over the Northern Atlantic,these have led to unrealistically large forecast concentrations or PM10 and PM2.5 and
poor performance of the daily analyses for ozone. Even with those problems however,LOTOS-EUROS was able to meet the RMSE reference targets for daily maxima andmeans in all occasions.
The bugs were corrected on the 1st of March 2016 and this expected to provideimproved performance for the coming quarters. A number of other improvements inthe model are also planned, including the implementation of data assimilation ofadditional ground-based observations and OMI NO2 for the daily analyses which are
also expected to contribute to an enhancement of the LOTOS-EUROS performance.
Qr. report on daily analyses and forecasts activities, verification of the LOTOS-EUROS performances | 19