regional production quarterly report on the daily …...lotos-euros; this issue was definitively...

Regional ProductionQuarterly report on the daily analyses and forecasts activities, and verification of the ENSEMBLE performances

December 2015 – January 2016 – February

2016

Issued by: METEO-FRANCE

Date: 04/05/2016

REF.:

CAMS50_2015SC1_D50.4.2-2016Q1_201605

CAMS50_2015SC1_D50.4.4-2016Q1_201605

CAMS50_2015SC1_D50.5.1-2016Q1_201605

Copernicus Atmosphere Monitoring Service

Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances |

This document has been produced in the context of the Copernicus Atmosphere Monitoring

Service (CAMS). The activities leading to these results have been contracted by the

European Centre for Medium-Range Weather Forecasts, operator of CAMS on behalf of the

European Union (Delegation Agreement signed on 11/11/2014). All information in this

document is provided "as is" and no guarantee or warranty is given that the information is

fit for any particular purpose. The user thereof uses the information at its sole risk and

liability. For the avoidance of all doubts, the European Commission and the European Centre

for Medium-Range Weather Forecasts has no liability in respect of this document, which is

merely representing the authors view.



Quarterly report on the dailyanalyses and forecasts activities, and verification ofthe ENSEMBLE performances

December 2015 – January 2016 – February

2016

METEO-FRANCE (M. Pithon, M. Plu, J. Parmentier,

J. Arteta, S. Guidotti, N. Assar)

Date: 04/05/2016

REF.:

CAMS50_2015SC1_D50.4.2-2016Q1_201605

CAMS50_2015SC1_D50.4.4-2016Q1_201605CAMS50_2015SC1_D50.5.1-2016Q1_201605


Contents:


1. Executive Summary.........................................................................................42. The ENSEMBLE................................................................................................5Product portfolio..................................................................................................5Availability statistics.............................................................................................5

3. Verification report............................................................................................8Verification of NRT forecasts.................................................................................8Verification of NRT analyses................................................................................13


1. Executive Summary

The Copernicus Atmosphere Monitoring Service (CAMS, www.copernicus-atmosphere.eu) is establishing the core global and regional atmospheric environmentalservice delivered as a component of Europe's Copernicus programme. The regionalforecasting service provides daily 4-days forecasts of the main air quality species and

analyses of the day before, from 7 state-of-the-art atmospheric chemistry models andfrom the median ensemble calculated from the 7 model forecasts. The regional servicealso provides posteriori reanalyses using the latest validated observation datasetavailable for assimilation.

This report covers the deliverables related to Near Real Time Production (NRT) for theENSEMBLE: D50.4.2-2016Q1, D50.4.4-2016Q1, D50.5.1-2016Q1, for the quarterDecember 2015 – January 2016 – February 2016. Verification is done against in-situ

surface observations. They are described in the report D50.1.2-2016Q1, that will bedelivered shortly. The verification of analyses is done against non assimilatedobservations.

During this quarter, the ENSEMBLE analyses and forecasts were produced on timeeveryday. The ENSEMBLE forecasts included at least 5 members everyday. TheENSEMBLE analyses included at least 5 members 99% of the time.

During this quarter, the ENSEMBLE scores reach their target (Key PerformanceIndicators) for every pollutants (O3, NO2, PM10, forecasts and analyses). Compared topast winter, the ENSEMBLE ozone forecasts show an improvement of the KPI score. ForNO2 and PM10 forecasts, the RMSE values are lower by about 1 µg.m-3 than during past

winter. These are the most noticeable improvements compared to past winter.Regarding the ENSEMBLE analyses, the diurnal patterns of the scores are similar forthe analyses and for the forecasts, but the analyses show a systematic improvement ofthe scores for all pollutants.

On-going model developments in every individual model, concerning theimplementation or refinement of secondary aerosol formation, and also the assimilationof new observed species at the surface, will help to improve ENSEMBLE scores in the

future.

Qr. report on daily analyses and forecasts activities, verification of the ENSEMBLE performances | 4


2. The ENSEMBLE

Product portfolio

Name Forecast Analysis

Description Forecast at surface, 50m,

250m, 500m, 1000m, 2000m,

3000m, 5000m above ground

Analysis at the surface

Available for users at 6:45 UTC for D0-D1

8:30 UTc for D2-D3

11:30 UTC for the day before

Species O3, NO2, CO, SO2, PM2.5, PM10,

NO, NH3, NMVOC, PANs,

Birch pollen at surface during

season

O3, NO2

Time span 0-96h, hourly 0-24h for the day before,

hourly

Availability statistics

The statistics below describe the ratio of days for which the ENSEMBLE model outputswere available on time to be included in the ENSEMBLE fields (analyses and forecasts)

that are computed at METEO-FRANCE. They are based on the following schedule for theprovision at METEO-FRANCE of:

- forecasts data before: 05:30 UTC for D0-D1 (up to 48h), 07:30 UTC for D2-D3 (from

49h to 96h)

- analyses data: before 11:00 UTC

These schedules have been set to meet the IT requirements for ENSEMBLE products

(no later than 8 UTC for 0-48h, 10 UTC for 49-96h and 12 UTC for analyses).



Indicators:

D0 D1 D2 D3

Schedule

compliance

base quarter DJF

ENSEMBLE Fcst

ENSEMBLE Ana

100%

100%

100%

100%

100%

100%

100%

100%

Number of model

forecasts

contributing to

ENSEMBLE

Dec 2015

Jan 2016

Feb 2016

N= 7: 84%

N>=5: 100%

N = 7: 94%

N>=5: 100%

N = 7: 86%

N>=5: 100%

N= 7: 84%

N>=5: 100%

N = 7: 87%

N>=5: 100%

N = 7: 83%

N>=5: 100%

N= 7: 52%

N>=5: 100%

N = 7: 77%

N>=5: 100%

N = 7: 79%

N>=5: 100%

N= 7: 29%

N>=5: 100%

N = 7: 77%

N>=5: 100%

N = 7: 69%

N>=5: 100%

Number of model

analyses

contributing to

ENSEMBLE

Dec 2015

Jan 2016

Feb 2016

N= 7: 58%

N>=5: 97%

N = 7: 81%

N>=5: 100%

N = 7: 90%

N>=5: 100%

Comments

During this quarter, all 7 models have contributed to the ENSEMBLE forecastcalculation in 86% of cases, for the first two day of forecasts (D0, D1). The lower score

for the next two days (64 % of cases with all 7 models for D2-D3) is improving at thesecond half of the quarter, thanks to the earlier provision of CHIMERE results.

Regarding analyses, except one day, the ENSEMBLE has always been calculated with at

least 5 members. During this quarter, in 76% of cases, all 7 models have effectivelybeen contributing.1

The poor figures for the December analyses were mainly due to the late provision of

LOTOS-EUROS; this issue was definitively solved mid December.

1 Statistics for analysis are calculated for O3 pollutant which is the only one provided by all models, at the

moment.



During this quarter, the following issues have been encountered by the ENSEMBLEproduction system:

Date Problem description

(origin, effects)

Impact

on production

08/12/2015 Internet issue at ECMWF:

internet dissemination, ec

access, Web services affected

Impact on ENSEMBLE

analysis: 2 models missing

11/12/2015 Issue related to the switch of

MATCH and EURAD-IM on

METEO-FRANCE operational

telecommunication system

Impact on ENSEMBLE

analysis: MATCH results

missing

Ensemble calculation with less than 5 models

Date Duration

(in days)

Missing models

08/12/2015 1 day: Analysis ENSEMBLE EURAD-IM, LOTOS-EUROS, MATCH

due to Internet problem at ECMWF

(EURAD-IM and MATCH missing) and

late arrival of LOTOS-EUROS

Total number of

days

1 analysis run



3. Verification report

This verification report covers the period December 2015 – January 2016 – February

2016. The ENSEMBLE skill scores are successively presented for four pollutants: ozone,NO2, PM10 and PM2.5. The skill is shown for the entire forecast horizon from 0 to 96h(hourly values), allowing to evaluate the entire diurnal cycle and the evolution ofperformance from day 0 to day 3. The forecasts and the analyses cover a large

European domain (25°W-45°E, 30°N-70°N). The statistical scores that are reported arethe root-mean-square error, the modified mean bias and the correlation. The surfaceobservations that are acquired by METEO-FRANCE and used for verification aredescribed in D50.1.2-2016Q1, that will be delivered shortly.

Verification of NRT forecasts

The following figures present, for each pollutant (ozone, NO2, PM10, PM2.5):

- in the upper-left panel, the root-mean square error of daily maximum (for ozone andNO2) or of daily mean (PM10) for the first-day forecasts with regards to surfaceobservations, for every quarter since DJF2014/2015, a target reference value isindicated as an orange line,

- in the upper-right panel, the root-mean square error of pollutant concentrationforecasts with regards to surface observations as a function of forecast term,

- in the lower-left panel, the modified mean bias of pollutant concentration forecastswith regards to surface observations as a function of forecast term,

- in the lower-right panel, the correlation of pollutant concentration forecasts withregards to surface observations as a function of forecast term.

The graphics show the performance of the ENSEMBLE (black curves).



ENSEMBLE forecasts: ozone skill scores against data from representativesites, period December 2015 - January 2016 - February 2016

The RMSE of the daily maximum of ozone ENSEMBLE forecasts reaches the target thisquarter. This score has improved by about 2 µg.m-3 compared to past winter. The biasis always positive as a function of the forecast hour and it is around 0 during theafternoon. The RMSE peaks at 19.5 µg.m-3 during night-time and gets as down as 15

µg.m-3 during daytime. The RMSE and correlation scores are similar as during pastwinter. There is a perceptible tendency to increasing RMSE and decreasing correlationwith forecast day.



ENSEMBLE forecasts: NO2 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

The RMSE of the daily maximum of NO2 ENSEMBLE forecasts reaches the target thisquarter. This score as a similar value as during past winter. The ENSEMBLE bias isalways negative, following a diurnal cycle with the highest absolute values duringdaytime and the lowest ones during night-time. The RMSE oscillates twice per day

between 9.5 and 18 µg.m-3, which is lower than during past winter. The correlation issimilar than during past winter and tends to decrease with forecast day.



ENSEMBLE forecasts: PM10 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

The RMSE of the daily mean of PM10 ENSEMBLE forecasts reaches the target this quar-ter. This score as a similar value as during past winter. The ENSEMBLE bias is alwaysnegative, but close to 0 in the late night. The RMSE oscillates between 10 µg.m-3 and15 µg.m-3, which is lower by about 1 µg.m-3 than during past winter. The correlation is

similar than during past winter and tends to decrease with forecast day.



ENSEMBLE forecasts: PM2.5 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

The RMSE of the daily mean of PM2.5 ENSEMBLE forecasts reaches the target thisquarter. This scores is lower than during past winter. The bias is mostly negative. TheRMSE is minimum during daytime (around 8.5 µg.m-3) and peaks at 11 µg.m-3 duringthe night. The correlation tends to decrease with forecast day.



Verification of NRT analyses

The following figures present, for each pollutant (ozone, NO2, PM10):

- in the upper-left panel, the root-mean square error of daily maximum (for ozone andNO2) or of daily mean (PM10) for the analyses (solid line) and for the first-day forecasts(dashed line) with regards to surface observations, for every quarter since

DJF2014/2015, a target reference value is indicated as an orange line,

- in the upper-right panel, the root-mean square error of pollutant concentration of theanalyses (solid line) and of the first-day forecasts (dashed line), with regards tosurface observations as a function of forecast term,

- in the lower-left panel, the modified mean bias of pollutant concentration forecasts ofthe analyses (solid line) and of the first-day forecasts (dashed line), with regards tosurface observations as a function of forecast term,

- in the lower-right panel, the correlation of pollutant concentration of the analyses

(solid line) and of the first-day forecasts (dashed line), with regards to surface observations as a function of forecast term.

The graphics show the performances of the ENSEMBLE (black curves). The

superposition of the analysis scores (solid lines) and of the forecast scores (dashedlines) computed over the same observation dataset is helpful to assess the added valueof data assimilation.



ENSEMBLE analyses: ozone skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

The RMSE of the daily maximum of ozone ENSEMBLE analyses reaches the target thisquarter. The diurnal patterns of the scores are similar for the analyses and for theforecasts, but the analyses show a clear improvement compared to the forecasts: byabout 0.8 for the mean bias, 4 µg.m-3 for the RMSE and 0.18 for the correlation.



ENSEMBLE analyses: NO2 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

The RMSE of the daily maximum of NO2 ENSEMBLE analyses reaches the target thisquarter. The diurnal patterns of the scores are similar for the analyses and for theforecasts, but the analyses show a clear improvement compared to the forecasts. Thebenefit of the assimilation is the higher during night-time.



ENSEMBLE analyses: PM10 skill scores against data from representative sites,period December 2015 - January 2016 - February 2016

ENSEMBLE production of PM10 analyses has not been implemented yet.



Analysis of ENSEMBLE performances for the quarter

The ENSEMBLE scores reach their target this quarter for every pollutant (O3, NO2, PM10,

forecasts and analyses).

The ozone RMSE and correlation scores are similar as during past winter, but the RMSEof daily ozone maximum has improved by about 2 µg.m-3 compared to past winter. For

the correlation of all pollutant forecasts, there is a perceptible decreasing tendencywith forecast day. The performance of ENSEMBLE ozone, NO2 and PM10 forecasts aresimilar compared to past winter, except for the ozone daily maximum which hasimproved significantly.

It was the first winter during which scores for the analyses have been reported. TheENSEMBLE analyses for O3 and NO2 perform significantly better than the ENSEMBLEforecasts for all the scores. The impact of assimilation of NO2 is the highest during

night-time.

On-going model developments in every individual model, concerning theimplementation or refinement of secondary aerosol formation, and also the assimilation

of new observed species at the surface, will help to improve ENSEMBLE scores in thefuture.


regional production quarterly report on the daily …...lotos-euros; this issue was definitively...

Documents