xi general cosmo meeting offenbach,7-10 september 2009

A.Montani; The COSMO-LEPS system: present status and outlookCOSMO meeting, Offenbach, 7-11 September 2009

The COSMO-LEPS system:

present status and outlook

Andrea Montani, Chiara Marsigli and Tiziana Paccagnella

ARPA-SIMC

Hydrometeorological Service of Emilia-Romagna, Bologna, Italy

XI General COSMO meetingOffenbach,7-10 September 2009


Outline

• Present status, recent updates, verification

• Future changes and plans


COSMO-LEPS suite @ ECMWF: present status

d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3

older EPSolder EPS

younger EPSyounger EPS

clustering clustering periodperiod

0000

1212

Cluster Analysis and RM identificationCluster Analysis and RM identification

4 variables4 variables

Z U V QZ U V Q

3 levels3 levels

500 700 850 hPa500 700 850 hPa

2 2 time time stepssteps


European European areaarea

Complete Complete LinkageLinkage

1616 Representative Representative Members driving the Members driving the 1616

COSMO-model COSMO-model integrations (weighted integrations (weighted according to the cluster according to the cluster

populations)populations)

employing either employing either Tiedtke or Kain-Fristch Tiedtke or Kain-Fristch

convection scheme convection scheme (randomly choosen) (randomly choosen)

++perturbations in perturbations in

turbulence schemeturbulence scheme

COSMO-LEPS

clustering area

• suite runs as a “time-critical application” managed by ARPA-SIMC;

• Δx ~ 10 km; 40 ML; fc+132h;• COSM0 v4.7 since Feb09

(with Runge-Kutta + multi-layer soil scheme);

• computer time (6.0 million BU for 2009) provided by the COSMO partners which are ECMWF member states.

COSMO-LEPS

Integration Domain


Recent upgrades

• 25 February 2009: – upgrade of COSMO to model version 4.7 with a few Meteoswiss code adaptation; use of “old” wind-gust diagnostic (itype_diag_gust=2);

• 24 May 2009:– beginning of experimental suite at 7 km with model version 4.8;

• 15 June 2009:– upgrade of INT2LM to model version 1.8 (no use of the “official executable” available at ECMWF, as ECMWF local definitions are not handled); use of ECMWF orography retrieved in spherical harmonics and no more on gaussian grid.


every now and then, “Floating-point exceptions” when the model writes output files over small subdomains …

Model explosion for COSMO-LEPS (last year)

Temporarily patch (which works, anyway!):the failed task is resubmitted with half time-step.

A more sophisticated investigation of model explosions will be carried out in the near future.

CURED by m

odel upgrade!


– SYNOP on the GTS

Objective verification of COSMO-LEPS

Main features:

variable: 12h cumulated precip (18-06, 06-18

UTC);

period : from Dec 2002 to Jul 2009;

region: 43-50N, 2-18E (MAP D-PHASEPHASE

area);

method: nearest grid point; no-weighted fcst;

obs: synop reports (about 470

stations/day);

fcst ranges: 6-18h, 18-30h, …, 102-114h, 114-126h;

thresholds: 1, 5, 10, 15, 25, 50 mm/12h;

system: COSMO-LEPS;

scores: ROC area, RPSS, Outliers, …

both monthly and seasonal scores were computed


Time series of ROC areaArea under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.

Poor performance of the system in Spring and Summer 2006 (both particularly dry), despite system upgrades.

Good performance during DOP 2007.

Small positive impact after 2007 system upgrades.

Scores in 2009 similar to those of last year (worsening for the highest threshold).


Seasonal scores of ROC area Performance of the system assessed for the last 5 summers (JJA) and the last 5 autumns

(SON).Consider the “event” 10 mm/12h; valuable forecast systems have ROC area values > 0.6.Need to take

into account the different statistics for each season (JJA 2004 and SON 2006 less rainy than the others).

Good scores during the full DOP in 2007.

Best performance for JJA and SON 2008, more evident for longer ranges.

Higher scores in SON than in JJA.


Time series of Brier Skill ScoreBSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if

BSS > 0.BS measures the mean squared difference between forecast and observation in probability

space. BS equivalent to MSE for deterministic forecast. The

improvement of model performance is detectable for all thresholds along the years.

Still problems with high thresholds for shorter ranges.

For the 78-90h range, a good trend has been evident since 2007: BSS has been positive for all thresholds in the last 12 months.


Ranked Probability Skill Score: time series + seasonal scores

A sort of BSS “cumulated” over all thresholds. RPSS is written as 1-RPS/RPSref. Sample climate is the reference system. RPS is the extension of the Brier Score to the multi-event situation.

Useful forecast systems, if RPSS > 0. Performance of the system assessed as time series and for the last 5 autumns (SON).

the improvement of the system performance is detectable for all forecast ranges along the years;

RPSS positive for all forecast ranges in 2007 and since May 2008.Best results for last autumn (almost no more 12-hourly cycle in the score).


Outliers: time series + seasonal scoresHow many times the analysis is out of the forecast interval spanned by the ensemble members. … the lower the better … Performance of the system assessed as time series and for summers 2004-2008.

Evident seasonal cycle (more outliers in winter), but overall reduction of outliers in the years up to 2007.

Reduction of outliers from one summer to the other, related to the increase of ensemble size (evident for the 10m to 16m increase).

Need to take into account the different statistics for each season.

Best results for the last 3 summers.


1) Implementation of COSMO-LEPS at 7 km

2) COSMO-LEPS for TIGGE-LAM

3) Multi-model clustering (“bridge-talk” WG4-CONSENSUS)

Present activity


Present systemx = 10 kmz = 40 MLt = 90 sngp = 306x258x40 = 3.157.920fcst range = 132hcost = 570 BU x runelapsed time = 25 min/runinitial conditions: interpolated

from EPS members

Implementation of COSMO-LEPS at 7 km

New system (COSMO-LEPS_7)

x = 7 kmz = 40 MLt = 60 sngp = 510x405x40 = 8.262.000fcst range = 132hcost = 2050 BU x runelapsed time = 48 min/runinitial conditions: interpolated

from EPS members merged with surface and soil-layer fields produced at DWD for COSMO-EU

COSMO-LEPS_7 running in experimental mode since 24 May 2009(no merging yet)


cleps_7 going operational in late Autumn

COSMO-LEPS 10 kmCOSMO-LEPS 7 km

the domain of COSMO-LEPS_7 is slightly larger than in the present configuration and fully covers Central and Southern Europe (British Isles and Iberia Peninsula included).

the grid points are in the locations as COSMO-EU grid.


COSMO-LEPS_10 (oper) vs COSMO-LEPS_7 (exp)

• Deterministic verification of T2M ensemble mean

Variable: 2-metre temperature. Period: 3 months, from 24/5 to 24/8/2009. Forecast ranges: fc+6h, fc+12h, …, fc+132h. Scores: root-mean-square error, bias.

• Probabilistic verification of 12-hour cumulated precipitation

Variable:12h cumulated precipitation (18-06, 06-18 UTC). Period: 2 months, June and July 2009. Forecast ranges: fc 6-18h, fc 18-30h, …, fc 114-126h. Scores: ROC area, BSS, RPSS, Outliers. Thresholds: 1, 5, 10, 15, 25, 50 mm/12h.

Observations: SYNOP reports over both MAP D-PHASE region (450 reports/day) and the FULL-DOMAIN (1400 reports/day).

Method: nearest grid point; no-weighted fcst.


Bias and rmse of T2M Ensemble Mean

Consider bias (the closer to zero, the better) and rmse (the lower the better) for June 2009. T2m forecasts are corrected with height.

For both models, slightly better scores over the FULL domain in comparison with the MAP D-PHASE domain.

Bias closer to zero and lower rmse for the 7-km suite. Improvement is not “massive”, but detectable for all forecast ranges, especially for day-time

verification. The signal is stable (similar scores for 1-month or 3-month verification).

MAPDOM

FULLDOM


ROC area, BSS, RPSS, OUTL for 12-hour tp (MAPDOM)

Consider the event “5 mm of precipitation in 12 hours” for ROC area and BSS in June 2009.

Slight better results for the 7-km suite, but the gap is very small (and significant?); outliers not reduced.

The improvement is detectable for all forecast ranges.Similar scores also July 2009 (less rainy) and for higher thresholds.

ROC area

BSS

% of outliers RPSS


ROC area, BSS, RPSS, OUTL for 12-hour tp (FULLDOM)

Consider the event “5 mm of precipitation in 12 hours” for ROC area and BSS in June 2009.

“Larger” (but still small) impact when verification is performed over the full domain.Need to consider also rainy seasons. The improvement is not “massive”, but is detectable for all forecast ranges.

ROC area

BSS

% of outliers RPSS


Products:• high-priority parameters (tp, t2m, td2m, u10, v10, gust10, mslp,

orog, lsm) operationally generated for each ensemble member from fc+0h to fc+132h every 3h;

• produced, in GRIB2 format, over a regular lat/lon grid (0.1x0.1);• saved at ARPA-SIMC; will be archived at ECMWF.

COSMO-LEPS for TIGGE-LAM

grib2 (TIGGE-LAM regular grid)

grib1 (original rotated grid)

rotated lat/lon domain regular lat/lon domain


Possible evolution scenarios

initial conditions by EPS

thesis by Andrea

Corigliano

Multi-model clustering (data from TIGGE portal)

initial conditions by MOGREPS

Consider both ECMWF EPS and UKMO MOGREPS and study properties of cluster analysis on multi-ensemble:

How many times do the 2 ensembles mix?Where do the RMs come from? How to they score depending on their “origin”?BEFORE DOWNSCSALING: is there added value with respect to single-model

ensemble?AFTER DOWNSCALING: is there added value with respect to single-model ensemble?

Cluster members chosen as representative members (RMs)


• COSMO-LEPS_7km:

– run the suite for ~ 40 more days and hope for larger impact !!!!!!

– use the soil moisture analysis fields provided by DWD or the fields provided by COSMO run in hindcast mode (“proxy-run”),

– implement it operationally by November.

• COSMO-LEPS for TIGGE-LAM:

– develop coding, post-processing and archiving of COSMO-LEPS output files in GRIB2 format;

– assistance to users.

Future plans

• Multi-model clustering: continue the work.

• Gaining from COSMO-SREPS experience, introduce more model perturbations.

• Support calibration and verification.


Thank you !


COSMO-LEPS (developed at ARPA-SIM)

• What is it?It is a Limited-area Ensemble Prediction System (LEPS),

based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).

• Why?It was developed to combine the advantages of global-

model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of severe and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)

generation of COSMO-LEPS to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of

severe weather events.


Operational set-up

Core products:16 perturbed COSMO-model runs (ICs and 3-

hourly BCs from 16 EPS members) to generate, “via weights”, probabilistic output: start at 12UTC; t = 132h;

Additional products: 1 deterministic run (ICs and 3-hourly BCs from the

high-resolution deterministic ECMWF forecast) to “join” deterministic and probabilistic approaches: start at 12UTC; t = 132h;

1 hindcast (or proxy) run (ICs and 3-hourly BCs from ECMWF analyses) to “downscale” ECMWF information: start at 00UTC; t = 36h.


but …


Score dependence on the domain size (1)

Verification of COSMO-LEPS against synop reports over the MAP D-PHASE area (~ 470 stations; MAPDOM) and the full domain (~ 1500 stations; fulldom):

different statistics of the verification samples; up to now, performance of the system over the 2 domains assessed only for 6

months (March-August 2007). difficult to draw general conclusions


Score dependence on the domain size (2)

RPSS

RPSS score… the higher the better… (and positive).

ROC area… the higher the better… (and above 0.6).Smoother transitions from month to month in “fulldom” scores.

Slightly better performance of COSMO-LEPS over the MAPDOM, but the signal varies from month to month.

Higher predictability with orographic forcing?

Need to check individual regions and/or to stratify for type of stations.

OUTL

ROC

Outliers percentage … the lower the better.


Bias and rmse of T2M Ensemble Mean

Consider bias (the closer to zero, the better) and rmse (the lower the better).

Bias closer to zero (0.5 °C of decrease) and lower rmse for the 7-km suite. The improvement is not “massive”, but detectable for all forecast ranges, especially for day-time

verification. The signal is stable (similar scores for 1-month or 3-month verification).Need to correct T2M forecasts with height to assess the impact more clearly.


Overestimation of Td2m and soil moisture (1)

Verification period: MAM07 and MAM08.

Obs: synop reports (about 470 stations x day).

Region: 43-50N, 2-18E (MAP D-PHASE area).

Larger bias and larger rmse in MAM08 rather than in MAM07 for COSMO-LEPS deterministic run (in 2007, no multi-layer soil model).


Overestimation of Td2m and soil moisture (2)

from Andrè Walser

COSMO-LEPS members have higher soil moisture than COSMO-7 run at Meteoswiss (case of 20/7), especially in the Alps.

For the future, do not simply interpolate from ECMWF IFS to COSMO model with the multi-layer soil model.

generate “home-made” soil moisture analysis (from COSMO-EU, from a proxy-run, …)


Semi-diurnal cycle in COSMO-LEPS scores

BSS score … the higher the better … Performance of the system assessed for 5 different Summers (JJA).

BSS Evident 12-hour cycle in

BSS scores (the same holds for RPSS, while less evident for ROC area scores).

Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …).

The amplitude of the cycle is somewhat reduced throughout the years and with increasing forecast range.

The bad performance in Summer 2006 is confirmed.


Feasibility study of COSMOLEPS at 7 km (cleps_7)

Motivations:Provide a more detailed description of

mesoscale processes by incresing the horizontal resolution.

Do not lose a “reasonable advantage” against ECMWF EPS, which will go to x=25 km during 2009.

“Keep the pace” with deterministic model (x~ 2-3 km): if the gap in resolutions between deterministic and probabilistic systems is too large, the two systems go for different solutions (that is, they forecast different weather!).

from 10 to 7 km (plus small domain extensions) does not seem a lot


Present systemx = 10 kmz = 40 MLt = 90 sngp = 306x258x40 =

3.157.920fcst range = 132hcost = 640 BU x runelapsed time = 45 min

COSMO-LEPS at 7 km (cleps_7): the answer to forecasters’ dream?

New systemx = 7 kmz = 40 MLt = 72 sngp = 510x405x40 =

8.262.000fcst range = 132hcost = 1925 BU x runelapsed time = 138 min

… cleps_7 is about 3 times more expensive than the present configuration

new computer at ECMWF being installed

Computer resources for each ECMWF member state will increase by a factor of 5 (five) and ….


The dream is possible

COSMO-LEPS 10 kmCOSMO-LEPS 7 km

the grid of cleps_7 would be almost identical to that of COSMO-EU, this making easier and cleaner the use of initial fields provided by DWD (e.g. soil moisture analysis).


Main results

• COSMO-LEPS system runs on a daily basis since November 2002 (6 “failures” in almost 6 years of activity) and it has become a “member-state time-critical application” at ECMWF ( ECMWF operators involved in the suite monitoring).

• COSMO-LEPS products used in EC Projects (e.g. PREVIEW), scientific experiments (e.g. COPS, MAP D-PHASE, EFAS) and met-ops rooms across COSMO community.

• Nevertheless, positive trends can be identified:• increase in ROC area scores and reduction in outliers percentages;• positive impact of increasing the population from 5 to 10 members (June

2004);• some deficiency in the skill of the system were identified after the system

upgrades occurred on Feb 2006 (from 10 to 16 members; from 32 to 40 ML + EPS upgrade!!!), but scores are encouraging throughout 2007 and, to a certain extent, 2008.

Time series scores cannot easily disentangle improvements related to COSMO-LEPS itself from those due to better boundaries by

ECMWF EPS.

2 more features:

• marked semi-diurnal cycle in COSMO-LEPS scores (better skill for “night-time” forecasts);

• better scores over the Alpine area rather than over the full domain (to be confirmed).


COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

tp > 1mm/24h

tp > 5mm/24h

Average values (boxes 0.5 x 0.5)MAM06

As regards AVERAGE precipitation above these two threshols, the 3 systems have similar performance.


ENSEMBLE SIZE REDUCTIONIMPACT EVALUATED ON CASE STUDIES (1)


2003082512 Friuli

(+72-+96)

20 mm 100 mm

5 RMs

10 RMs

All 51


ENSEMBLE SIZE REDUCTIONIMPACT EVALUATED ON CASE STUDIES (2)

Observed precipitation between 15-11-2002 12UTC and 16-11-2002 12 UTC

Piedmont case


2002111212 Piedmont

20 mm 150 mm

5 RMs

10 RMs

All 51


COSMO-LEPS

Real time products


Why Limited Area Ensemble Prediction?

• Global Ensemble Prediction Systems

– have become extremely important tools to tackle the problem of predictions beyond day 2

– are usually run at a coarser resolution with respect to deterministic global predictions → skill in forecasting intense and localised events is currently still limited.


Why Limited Area Ensemble Prediction?(2)

As regards high resolution deterministic forecast in the short range, where limited-area models play the major role, a “satisfactory” QPF is still one of the major challenges. The same can be said for other local parameters.

This is due, among other reasons, to the inherently low degree of predictability typical of severe and localised events.

Probabilistic/Ensemble approach is so required also for the short range at higher resolution


From Global EPS to LAM EPS

• In the Limited-area ensemble systems, tailored for the short range, perturbations must be already “active” during the first hours of integration

• The characteristic of the LAM ensemble are strongly dependent by the lateral boundaries forcing.

• Due to the “regional” application of these Limited Area Ensembles, methodologies can be different in different geographycal regions.

A pratical consideration:Global EPS ~ Big Centres

Limited Area EPS ~ (also) Relatively Small Centres


In the last period the verification package is being developed keeping into account two measure of precipitation:

the cumulative volume of water deployed over a specific region

the rainfall peaks which occur within this region

OBJECTIVE VERIFICATION OF COSMO-LEPS

COSMO observations


CLEPS EPS

Verification grid

OBS MASK


COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

tp > 1mm/24h

tp > 5mm/24h

Average values boxes 0.5x0.5 deg3 sis


COSMO-LEPS

16-MEMBER EPS

tp > 1mm/24h

NOCC=610

NOCC=1195

tp > 5mm/24h

NOCC=2671

tp > 10mm/24h

ave 0.5


Maximum values boxes 0.5x0.5 deg

tp > 1mm/24h

tp > 5mm/24h

tp > 10mm/24h

COSMO-LEPS

16-MEMBER EPS

51-MEMBER EPS

3 sis


COSMO-LEPS

16-MEMBER EPS

tp > 20mm/24hNOCC=227

Average values boxes 0.5x0.5 deg


COSMO-LEPS NW

COSMO-LEPS W

tp > 1mm/24h

COSMO-LEPS weighting procedure

maximum values (boxes 0.5x0.5 deg)


The COSMO-LEPS suite @ ECMWFNovember 2002 – May 2004

d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3

oldest EPSoldest EPS

1212

middle EPSmiddle EPS

oldest EPSoldest EPS

youngest EPSyoungest EPS


0000

1212



Z U V QZ U V Q

3 levels3 levels

500 700 850 hPa500 700 850 hPa





5 Representative 5 Representative MembersMembers

Driving the 5Driving the 5COSMO-model COSMO-model integrationsintegrations

COSMO-LEPS

Integration Domain

COSMO-LEPS

clustering area


The COSMO-LEPS suite @ ECMWFJune 2004 – January 2006

d-1d-1 dd d+5d+5d+1d+1 d+2d+2 d+4d+4d+3d+3

middle EPSmiddle EPS

youngest EPSyoungest EPS


0000

1212



Z U V QZ U V Q

3 levels3 levels

500 700 850 hPa500 700 850 hPa





COSMO-LEPS

Integration Domain

10 10 Representative Representative

MembersMembersdriving the 10driving the 10COSMO-model COSMO-model integrationsintegrations

employing either employing either Tiedtke or Kain-Tiedtke or Kain-Fristch scheme Fristch scheme

randomly randomly choosenchoosen

COSMO-LEPS

clustering area

• Suite running in real time at ECMWF managed by ARPA-SIM;

• Δx ~ 10 km• Fc length: 120h


COSMO-LEPS (developed at ARPA-SIM)

• What is it?It is a Limited-area Ensemble Prediction System

(LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).

• Why?Because the horizontal resolution of global-model

ensemble systems is limited by computer time constraints and does not allow a detailed description of mesoscale and orographic-related processes.

The forecast of heavy precipitation events can still be inaccurate (in terms of both locations and intensity) after the short range.


COSMO-LEPS project

combine the advantages of global-model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of intense and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)

generation of COSMO-LEPS in order to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of the so-

called “severe weather events”.


Dim 2

Initial conditions Dim 1 Dim 2

Possible evolution scenarios

Dim 1 Initial conditions

ensemble size reduction

Cluster members chosen as representative members (RMs)

LAM integrations driven byRMs

LAM scenario

LAM scenario

LAM scenario

COSMO-LEPS methodologyCOSMO-LEPS methodology


Semi-diurnal cycle in COSMO-LEPS scores

BSS score … the higher the better … Performance of the system assessed for 5 different Summers (JJA).

BSS Evident 12-hour cycle in

BSS scores (the same holds for RPSS, while less evident for ROC area scores).

Better performance of the system for “night-time” precipitation, that is for rainfall predicted between 18Z and 6Z (ranges 30-42h, 54-66h, …).

The amplitude of the cycle is somewhat reduced throughout the years and with increasing forecast range.

The bad performance in Summer 2006 is confirmed.


Main results

• COSMO-LEPS system runs on a daily basis since November 2002 (6 “failures” in almost 5 years of activity) and it has become a “member-state time-critical application” at ECMWF ( ECMWF operators involved in the suite monitoring).

• COSMO-LEPS products used in EC Projects (e.g. PREVIEW) , field campaigns (e.g. COPS, MAP D-PHASE) and met-ops rooms across COSMO community.

• Nevertheless, positive trends can be identified:• increase in ROC area scores and reduction in outliers percentages;• positive impact of increasing the population from 5 to 10 members (June

2004);• although some deficiency in the skill of the system were identified after the

system upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels + EPS upgrade!!!), scores are encouraging throughout 2007.

Time series scores cannot easily disentangle improvements related to COSMO-LEPS itself from those due to better boundaries by

ECMWF EPS.

2 more features:

• marked semi-diurnal cycle in COSMO-LEPS scores (better skill for “night-time” forecasts);

• better scores over the Alpine area rather than over the full domain (to be confirmed).


4 talks per il COSMO meeting

• Parallel session:a) COSMO-LEPS: status and plans (praticamente ok)

b) application of COSMO-LEPS in PREVIEW project (pronta)

c) Multi-model clustering for dynamical downscaling (in alto mare)

• Plenary session:– COSMO-LEPS: status and plans riassuntino di a)

xi general cosmo meeting offenbach,7-10 september 2009

Documents

cosmoleps system

upgrade of cosmo

cosmomodel integrations

model upgrade

model version

cosmo partners

system upgrades

good scores