ecmwf - t. haiden, m. janousek, p. bauer, j. bidlot, m. dahoui, l. … · 2015. 12. 14. ·...

765

Evaluation of ECMWF forecasts,

including 2014-2015 upgrades

T. Haiden, M. Janousek, P. Bauer, J. Bidlot, M. Dahoui, L. Ferranti, F. Prates,

D.S. Richardson and F. Vitart

Research and Forecast Department

November 2015

Series: ECMWF Technical Memoranda A full list of ECMWF Publications can be found on our web site under:

http://www.ecmwf.int/en/research/publications

Contact: [email protected] © Copyright 2015 European Centre for Medium Range Weather Forecasts Shinfield Park, Reading, Berkshire RG2 9AX, England Literary and scientific copyrights belong to ECMWF and are reserved in all countries. This publication is not to be reprinted or translated in whole or in part without the written permission of the Director. Appropriate non-commercial use will normally be granted under the condition that reference is made to ECMWF. The information within this publication is given in good faith and considered to be true, but ECMWF accepts no liability for error, omission and for loss or damage arising from its use.

EvaluationofECMWFforecastsincluding2014‐2015upgrades

TechnicalMemorandumNo.765 1

1. IntroductionRecentchangestotheECMWFforecastingsystemaresummarisedinsection2.VerificationresultsoftheECMWFmedium‐rangeupper‐airforecastsarepresentedinsection3,including,whereavailable,acomparisonofECMWF’sforecastperformancewiththatofotherglobalforecastingcentres.Section4presentstheverificationofECMWFforecastsofweatherparametersandoceanwaves,whilesevereweatherisaddressedinsection5.Finally,section6discussestheperformanceofmonthlyandseasonalforecastproducts.

Atits42ndSession(October2010),theTechnicalAdvisoryCommitteeendorsedasetoftwoprimaryandfoursupplementaryheadlinescorestomonitortrendsinoverallperformanceoftheoperationalforecastingsystem.Theseheadlinescoresareincludedinthecurrentreport.Asinpreviousreportsawiderangeofcomplementaryverificationresultsisincludedand,toaidcomparisonfromyeartoyear,thesetofadditionalverificationscoresshownhereismainlyconsistentwiththatofpreviousyears(ECMWFTech.Memos.346,414,432,463,501,504,547,578,606,635,654,688,710,742).Ashorttechnicalnotedescribingthescoresusedinthisreportisgivenintheannextothisdocument.

VerificationpageshavemostlybeenmovedtothenewECMWFwebsiteandareregularlyupdated.Theyareaccessibleatthefollowingaddress:

www.ecmwf.int/en/forecasts/charts

bychoosing‘Verification’undertheheader‘MediumRange’

(medium‐rangeandoceanwaves)

bychoosing‘Verification’undertheheader‘ExtendedRange’

(monthly)

bychoosing‘Verification’and‘Seasonalforecasts‘undertheheader‘LongRange’

(seasonal)

2. ChangestotheECMWFforecastingsystemInNovember2014,ECMWFimplementedanintermediatecycle(40r1.1)ofitsIntegratedForecastingSystem(IFS)thataddedthecapabilitytoactivelyassimilateallconventionalobservationaldatainBUFRformat(binarycode).ThismodificationwasneededsinceWMOallowedproviderstostopdisseminatingdatainTraditionalAlphanumericCodes(TAC)formatinNovember2014.WMOdecidedtomovetoarepresentationofobservationsinBUFRbecauseofthelackofflexibilityoftheTACformatusedfortheexchangeofsurfaceandupperairobservationsforthelast50years.ThenewBUFRformatallowssubstantiallymoredatatobeprovided,forexamplemuchhigherverticalresolutioninradiosondereports,thanwaspossiblewiththeTACformat.However,asignificantandcontinuingefforthasbeennecessarytomonitorthetransitiontothenewformatasnumerouserrorsandqualityissueshavebeenidentifiedasdataprovidersintroducethechange.ECMWFhasbeenplayinganactiveroleinthiseffortinclosecollaborationwiththeMemberStates,EUMETNET’sObservationsProgrammeandtheWMO.

Anewmodelcycle(41r1)wasimplementedon12May2015.


2 TechnicalMemorandumNo.765

Cycle41r1introducesalakeparametrization,basedontheFLakemodel,whichisappliedtoallresolvedandsub‐gridscalelakes.Theworkonlakesresultinginthisimplementationhasbenefittedfromamulti‐yearcollaborationwiththelake‐NWPcommunityinEuropeandinparticularrecognizesthescientificco‐ordinationoftheDeutscheWetterdienst.Thisimplementationimproves2‐metretemperatureforecastsinthevicinityofsmalllakesandnearcoastlinesnotrepresentedinthepreviousmodel.

Oceanwaveforecastsbenefitfromtheextensionofthehigh‐resolutionwavemodelfromtheEuropeanandNorthAtlanticregiontothewholeoftheglobe.Thesestand‐aloneforecastsaredrivenbythehighresolutionforecastHRES,areperformedatahigherresolutionthanthecoupledwavemodel,andincludeaforcingbyoceancurrents.

Newland‐seamask,orographyandclimatefields(glacierinformation,surfacealbedo)havebeenintroduced,aswellasnewdataforlakedepthandotherlakeparameters.ThenewmodelalsousesnewCO2,O3andCH4climatologiesfromthelatestMACC‐IIreanalysis.

Arevisedverticalinterpolationinthesemi‐Lagrangianadvectionschemereducesgravitywavenoiseduringsuddenstratosphericwarmingevents.

Theinner‐loopresolutionsofthe4DVARdataassimilationsystemhavebeenupgradedtoT255(80km)foreachofthethreeiterationsoftheouterloopstoproducefinerscaleincrements.Thebackgrounderrorcovariancesaremademoreflow‐dependentbyreducingthesamplingwindowandaveragingthestatisticsovershorterpastperiods,thesedynamicalstatisticsbeingusedjointlywithaclimatology.Arangeofadditionalsatelliteobservationsimprovestherepresentationoflandsurface,seaiceandoceanwaveparameters.

Monthlyensembleforecastsandre‐forecastshavebeenextendedfrom32to46days.Theextendedforecastsshouldbeusedwithcarebutresultshaveshownthatthereispositiveskillinsomeaspectsofforecastsinthe30–46dayrange.Theensembleforecast(ENS)re‐forecastdatasetissignificantlyenhanced,withre‐forecastsrunningtwiceaweek,forMondaysandThursdays(previouslyjustThursdays),andwiththesizeofeachre‐forecastensembleincreasedfrom5to11members.Thisprovidesasubstantialincreaseinthesamplesizeforthemodelclimatesforthemedium‐rangeExtremeForecastIndex(EFI)/ShiftofTails(SOT)andtheextended‐range(monthly)forecastanomalyproducts.

AsummaryofforecastperformanceisprovidedasascorecardinFigure1.

Thenewmodelcycleimprovesbothhigh‐resolutionforecasts(HRES)andensembleforecasts(ENS)throughoutthetroposphereandinthelowerstratosphere.Improvementsareseenbothinverificationagainstthemodelanalysisandverificationagainstobservations.

Cycle41r1bringsconsistentgainsinforecastperformanceatthesurfacefortotalcloudcoverandprecipitation.Improvementsinthemodellingofcloudandprecipitationreducethepredictedoccurrenceofdrizzleinsituationswherelarge‐scaleprecipitationdominates,andtheyincreasetheamountofrainfallinforecastsofintenseevents,leadingtoabettermatchwithobservations.Improvementsarealsoseenfor2‐metretemperatureand2‐metrehumidityinpartsofthenorthernhemisphereandthetropics.Cycle41r1alsointroducesanumberofnewoutputparameters,suchasprecipitationtype,includingfreezingrain.

Theaveragepositionerrorfortropicalcyclonesisslightlyreduced,andtropicalcyclonesaregenerallyforecasttobemoreintense.Forexample,IFSCycle41r1performedbetterthanCycle40r1inpredictingthetrackoftropicalcyclonePam,whichdevastatedVanuatuintheSouth



PacificinMarch2015.InHRES,thesealevelpressureminimumatthecentreoftropicalcyclonesisonaverageslightlyloweratallleadtimes.Uptoandincludingday3thismakestheforecastbetter,byreducingtheslightpositivebias.Fromday5onwards,however,thepre‐existingbiastowardsover‐deepeninghasincreasedslightly.

Thenewmodelcycleisdescribedingreaterdetailat

http://www.ecmwf.int/en/forecasts/documentation‐and‐support/changesecmwf‐model/cycle‐41r1.

3. Verificationforupper‐airmedium‐rangeforecasts

3.1. ECMWFscoresFigure2showstheevolutionoftheskillofthehigh‐resolutionforecastof500hPaheightoverEuropeandtheextratropicalnorthernandsouthernhemispheressince1981.Eachpointonthecurvesshowstheforecastrangeatwhichthemonthlymean(bluelines)or12‐monthmeancentredonthatmonth(redline)oftheanomalycorrelation(ACC)betweenforecastandverifyinganalysisfallsbelow80%.InbothhemispheresandoverEuropescoreshavebeenconsistentlyhigh.Resulting12‐monthmeansareequaltoorslightlyexceedingthehighestpreviousvalues.

Acomplementarymeasureofperformanceistherootmeansquare(RMS)erroroftheforecast.Figure3showsRMSerrorsforbothextratropicalhemispheresofthesix‐dayforecastandthepersistenceforecast.Theerrorofthesix‐dayforecasthasfurtherdecreasedinthehemisphericaverages.

Figure4showsthetimeseriesoftheaverageRMSdifferencebetweenfour‐andthree‐day(blue)andsix‐andfive‐day(red)forecastsfromconsecutivedaysof500hPaforecastsoverEuropeandthenorthernextratropics.Thisillustratestheconsistencybetweensuccessive12UTCforecastsforthesameverificationtime;thegeneraldownwardtrendindicatesthatthereisless“jumpiness”intheforecastfromdaytoday.Thelevelofconsistencybetweenconsecutiveforecastshasincreasedfurtherinthelastyear.In2014the12‐monthmovingaveragesofRMSdifferencesreachedtheirlowestvaluessofar.

ThequalityofECMWFforecastsfortheupperatmosphereinthenorthernhemisphereextratropicsisshownthroughtimeseriesoftemperatureandwindscoresat50hPainFigure5.Scoresforone‐dayforecastsoftemperatureaswellasforecastsofvectorwindhavebeenstablesincelastyear.

Theverificationofmodelforecastsinthestratosphereiscurrentlyperformedagainstanalysesandradiosondeobservations.Bothdatasetshavelimitationsthatreduceourabilitytoproperlyassessmodeldevelopmentsinthestratosphere.GPSradiooccultation(RO)observationsrepresentanalternativewayofverificationinthestratosphere.Theyhaveagoodverticalresolution,globalandhomogeneousdistribution(around3000profilesperday),andtheydonotrequirebiascorrection.

SinceROmeasurementsareassimilateddirectlyintheformofbendingangles,oneoptionistoperformtheverificationusingthisquantity,whichisprimarilysensitivetovariationsintemperatureinthestratosphere.RObendinganglescanprovidearobustmeasureofforecasterrorchangesbetweenmodelcycles.Figure6(topleftpanel)showsthereductioninerrorstandarddeviationofbendinganglesduetothemostrecentmodelupgrade.Thebottompanel



showsthecorrespondingtime‐seriesofbendingangledeparturesforbothmodelversions,indicatingthatthelargestreductionsintherandompartoftheforecasterroroccurduringstratosphericwarmingevents.

Sinceverificationresultsforbendinganglescanbedifficulttointerpret,temperatureretrievalsfromGPS‐ROrepresentanalternativewayofassessingtheimpactofmodelchangesonsystematictemperatureforecasterrorsinthelowerandmiddlestratosphere.However,GPS‐ROtemperatureretrievalsrequireaprioriinformationabouttheupperatmosphere.InordertohaverobustresultswhencomparingIFScyclesitisimportanttousethesamepriorinformationforROtemperatureretrievals,whichinthiscaseisprovidedbytheoperational6‐hourforecast.TheupperrightpanelinFigure6showsthereductioninthestandarddeviationofthetemperaturedeparturescorrespondingtotheimprovementinbendingangledeparturesshownintheleftpanel.

ThetrendinENSperformanceisillustratedinFigure7,whichshowstheevolutionofthecontinuousrankedprobabilityskillscore(CRPSS)for850hPatemperatureoverEuropeandthenorthernhemisphere.AsforHRES,theENSskillreachedrecordlevelsinwinter2009–10.Therehasbeensomereductionfromtheserecordlevels,especiallyoverEurope,asmightbeexpectedandaswasseenalsoforHRES.However,theENSperformancehasbeenconsistentlyhigh,andtheskillinwinter2013–14inthenorthernextratropicshasbeenverysimilartotherecordlevelsof2010.Anumberofchangeshavebeenmadetotheensembleconfigurationsince2010,includingimprovementstoboththeinitialperturbationsandrepresentationofmodeluncertainties,theincreaseinresolutioninJanuary2010,andfurtherredefinitionofperturbationsusingtheensembleofdataassimilations.Theslightlydecreasingtrendin2014isduetoatmosphericvariability.

Inawell‐tunedensemblesystem,theRMSerroroftheensemblemeanforecastshould,onaverage,matchtheensemblestandarddeviation(spread).Theensemblespreadandensemble‐meanerrorovertheextratropicalnorthernhemisphereforlastwinter,aswellasthedifferencebetweenensemblespreadandensemble‐meanerrorforthelastthreewinters,areshowninFigure8.Thematchbetweenspreadanderrorin2014issimilartopreviousyears,althoughslightlystrongerunder‐dispersioncanbeseeninthemediumrange.Theunder‐dispersionfortemperatureat850hPainbothseasonsisstillpresent,althoughuncertaintyintheverifyinganalysisshouldbetakenintoaccountwhenconsideringtherelationshipbetweenspreadanderrorinthefirstfewdays.

Agoodmatchbetweenspatiallyandtemporallyaveragedspreadanderrorisanecessarybutnotasufficientrequirementforawell‐calibratedensemble.Itshouldalsobeabletocaptureday‐to‐daychanges,aswellasgeographicalvariations,inpredictability.Thiscanbeassessedusingspread‐reliabilitydiagrams.Forecastvaluesofspreadoveragivenregionandtimeperiodarebinnedintoequallypopulatedspreadcategories,andforeachbintheaverageerrorisdetermined.Inawell‐calibratedensembletheresultinglineshouldbeclosetothediagonal.Figure9andFigure10showspread‐reliabilityplotsfor500hPageopotentialand850hPatemperatureinthenorthernextratropics(top),Europe(centre),andthetropics(bottom,inFigure10only)fordifferentglobalmodels.Spreadreliabilitygenerallyimproveswithleadtime.Atday1(leftpanels),forecaststendtobemorestronglyunder‐dispersiveatlowspreadvaluesthanatday6(rightpanels).ECMWFperformsverywell,withitsspreadreliabilityusuallyclosesttothediagonal.Thestarsintheplotsmarktheaveragevalues,correspondingtoFigure



8,andideallyshouldlieonthediagonal,ascloselyaspossibletothelowerleftcorner.Alsointhisrespect,ECMWFperformsbestoverall.

InordertohaveabenchmarkfortheENS,theCRPShasbeencomputedfora‘dressed’ERA‐I.ThisalsohelpstodistinguishtheeffectsofIFSdevelopmentsfrompureatmosphericvariability.Thedressingusesthemeanerrorandstandarddeviationoftheprevious30daystogenerateaGaussiandistributionaroundtheERA‐I.Figure11showstheevolutionoftheCRPSfortheENSandforthedressedERA‐Ioverthelast10yearsfortemperatureat850hPaatforecastday5.InthenorthernhemispheretheskilloftheENSrelativetothereferenceforecastwasabout15%in2005andisapproaching30%in2015.ItisworthnotingthatusingtheforecasterrorfordressingoftheERA‐Iisequivalenttogeneratinganearlyperfectlycalibratedensemble.Thusthissortofreferenceforecastrepresentsachallengingbenchmark.

Theforecastperformanceoverthetropics,asmeasuredbyRMSvectorerrorsofthewindforecastwithrespecttotheanalysis,isshowninFigure12.At200hPa(upperpanel)the1‐dayforecasthascontinuedtoimproveslightly(althoughitisstillslightlyhigherthantheminimumwhichwasreachedin2003–2004),whilethe5‐dayforecasterrorhasincreased.Similarly,at850hPa(lowerpanel)theerroratday1hasbeenslightlyreducedwhileatday5ithasincreased.Theincreaseat850hPaisalsoseeninERA‐Interim(notshown)andinforecastsofothercentres.Itoccursforverificationagainstanalysisanddoesnotappearwhentheforecastisverifiedagainstobservations(cf.Section3.2,Figure16andFigure17).Notethatscoresforwindspeedinthetropicsaregenerallysensitivetointer‐annualvariationsoftropicalcirculationsystemssuchastheMadden‐Julianoscillation,orthenumberoftropicalcyclones.

3.2. WMOscores‐comparisonwithothercentresThecommongroundforcomparisonistheregularexchangeofscoresbetweenWMOdesignatedglobaldata‐processingandforecastingsystem(GDPFS)centresunderWMOcommissionforbasicsystems(CBS)auspices,followingagreedstandardsofverification.Thenewscoringproceduresforupper‐airfieldsusedintherestofthisreportwereapprovedforuseinthisscoreexchangebythe16thWMOCongressin2011andarenowbeingimplementedatparticipatingcentres.ECMWFceasedcomputationofscoresusingpreviousproceduresinDecember2011.ThereforetheECMWFscoresshowninthissectionareacombinationofscoresusingtheold(untilDecember2011)andnewprocedures(from2012onward).Thescoresfromothercentresfortheperiodofthisreporthavebeencomputedstillusingthepreviousprocedures.ForthescorespresentedheretheimpactofthechangesisrelativelysmallfortheECMWFforecastsanddoesnotaffecttheinterpretationoftheresults.

Figure13showstimeseriesofsuchscoresfor500hPageopotentialheightinthenorthernandsouthernhemisphereextratropics.Overthelast10yearserrorshavedecreasedforallmodels,especiallyduringthewinterseason.ECMWFcontinuestomaintainaleadovertheothercentres.

WMO‐exchangedscoresalsoincludeverificationagainstradiosondesoverregionssuchasEurope.Figure14(Europe),andFigure15(northernhemisphereextratropics)showingboth500hPageopotentialheightand850hPawindforecasterrorsaveragedoverthepast12months,confirmsthegoodperformanceoftheECMWFforecastsusingthisalternativereferencerelativetotheothercentres.

ThecomparisonforthetropicsissummarisedinFigure16(verificationagainstanalyses)andFigure17(verificationagainstobservations).Whenverifiedagainstthecentres’ownanalyses,



theJapanMeteorologicalAgency(JMA)forecasthasthelowesterrorintheshortrange(day1)whileinthemediumrange,ECMWFandJMAaretheleadingmodelsinthetropics.Atthebeginningof2012theerrorsoftheECMWFforecastat850hPahaveshiftedtoaslightlylowerlevelduetoachangeinthecomputationofthescore.Insteadofsamplingthefullfieldsona2.5°grid,fieldsarenowspectrallytruncatedequivalentto1.5°resolution,inaccordancewithWMOguidelines.Inthetropics,verificationagainstanalyses(Figure16)isverysensitivetotheanalysis,inparticularitsabilitytoextrapolateinformationawayfromobservationlocations.Whenverifiedagainstobservations(Figure17),theECMWFforecasthasnowthesmallestoverallerrorsbothintheshortandmediumranges.

4. Weatherparametersandoceanwaves

4.1. Weatherparameters–high‐resolutionandensembleThesupplementaryheadlinescoresfordeterministicandprobabilisticprecipitationforecastsareshowninFigure18.Thetoppanelshowstheleadtimeatwhichthestableequitableerrorinprobabilityspace(SEEPS)skillforthehigh‐resolutionforecastforprecipitationaccumulatedover24hoursovertheextratropicsdropsbelow45%.Thisthresholdhasbeenchosensuchthatthescoremeasurestheskillataleadtimeof3–4days.ThebottompanelshowstheleadtimeatwhichtheCRPSSfortheprobabilityforecastofprecipitationaccumulatedover24hoursovertheextratropicsdropsbelow10%.Thisthresholdhasbeenchosensuchthatthescoremeasurestheskillataleadtimeofapproximately6days.Bothscoresareverifiedagainststationobservations.

MuchoftherecentvariationofthescoreforHRESisduetoatmosphericvariability,asshownbycomparisonwiththeERA‐Interimreferenceforecast(dashedlineinFigure18,toppanel).BytakingthedifferencebetweentheoperationalandERA‐Interimscoresmostofthisvariabilityisremoved,andtheeffectofmodelupgradesisseenmoreclearly(centrepanelinFigure18).Whilethelargestimprovementisassociatedwiththeintroductionofthefive‐speciesmicrophysicsinNovember2011(cycle36r4),microphysicschangesinsubsequentcyclesledtoafurtherincreaseinskill.Theprobabilisticscore(lowerpanelinFigure18)showssomerecentimprovementafterthestagnantperiod2010–2012whichwasagainpartlyduetoatmosphericvariability.TheCRPSoftheclimatologyforecast,whichisusedasareferencefortheCRPSS(seeAppendixA.2),decreased(i.e.improved)overtheperiod2010–2011,whichhasmaskedimprovementsduetomodelupgradesduringthattime.In2012,however,thistrendhasreversed,sothatmodelimprovementshavebecomemorevisibleagainintheCRPSS.

ECMWFperformsaroutinecomparisonoftheprecipitationforecastskillofECMWFandothercentresforboththehigh‐resolutionandtheensembleforecastsusingtheTIGGEdataarchivedintheMeteorologicalArchivalandRetrievalSystem(MARS).Resultsusingthesesameheadlinescoresforthelast12monthsshowtheHRESleadingwithrespecttotheothercentresfromday3onwardswhiletheMet‐Officemodelisleadingatday1(Figure19,upperpanel),andfortheENSaconsistentclearleadforECMWFoverthewholeleadtimerange(Figure19,bottompanel).

Trendsinmeanerrorandstandarddeviationoverthelast10yearsoferrorfor2mtemperature,2mdewpoint,totalcloudcover,and10mwindspeedforecastsoverEuropeareshowninFigure20toFigure23.VerificationisagainstsynopticobservationsavailableontheGlobalTelecommunicationSystem(GTS).Acorrectionforthedifferencebetweenmodel



orographyandstationheightwasappliedtothetemperatureforecasts,butnootherpost‐processinghasbeenappliedtothemodeloutput.

Ingeneral,theperformanceoverthepastyearfollowsthetrendofpreviousyears.For2mtemperatureanddewpoint,theerrorstandarddeviation(uppercurvesineachplot)hasbeencomparativelysmall.However,negativebiaseswithmarkedannualcyclespersist,especiallyatnight‐time.Fortotalcloudcover(Figure22)thebiashasbeensmallinrecentyears,andtheerrorstandarddeviationhasshownlittlechange.Forwindspeed(Figure23)thereducedlevelofnight‐timebiasassociatedwiththechangeinsurfaceroughnessinNovember2011hasbeenmaintained.Howeverthedaytimebiashasbecomeslightlymorenegative.

Tocomplementtheevaluationofsurfaceweatherforecastskill,resultsobtainedforverificationagainstthetopoftheatmosphere(TOA)reflectedsolarradiationproducts(dailytotals)fromtheClimateMonitoringSatelliteApplicationFacility(CM‐SAF)basedonMeteosatdataareshown.Thereisanincreaseintheskilloftheoperationalhigh‐resolutionforecastrelativetoERA‐Interiminrecentyears,bothintheextratropicsandtropics(Figure24),thatcanbeattributedtothecombinedeffectofaseriesofmodelchangesbeginningwiththeintroductionofthefive‐speciesprognosticmicrophysicsschemeinNovember2010(cycle36r4).

ERA‐InterimisusefulasareferenceforecastfortheHRESasitallowsfilteringoutmuchoftheeffectofatmosphericvariationsonscores.Figure25showstheevolutionofskillatday5relativetoERA‐Interiminthenorthernhemisphereextratropicsforvariousupper‐airandsurfaceparameters.ThemetricusedistheRMSEforupper‐airfieldsandtheerrorstandarddeviationforthesurfacefields.Curvesshow12‐monthrunningmeanvalues.Itcanbeseenthatthelargestrelativeimprovements(15–20%since2002)havebeenachievedforupper‐airanddynamicfields,followedby2mtemperatureand10mwindspeed.Theskilloftotalcloudcover,whichhadbeenstablepriorto2011,startedtoincreaseasaresultofmorerecentcyclechanges.Formeansealevelpressure,thehighest12‐monthskillsofarwasreachedattheendof2014.Bothfor500hpageopotentialand850hPatemperature,maximasofaroccurredearlyin2014,andfurtherincreasesareexpectedfromthemodelupgradeinMay2015.

4.2. OceanwavesThequalityoftheoceanwavemodelanalysisandforecastisshowninthecomparisonwithindependentoceanbuoyobservationsinFigure26.ThetoppanelofFigure26showstimeseriesoftheforecasterrorfor10mwindspeedusingthewindobservationsfromthesebuoys.Theforecasterrorhassteadilydecreasedsince2001andithasreacheditslowestvaluesofarinthewinterseason2013–14.Errorsinthewaveheightforecastin2014–15havebeenthelowestsofarinthe1–3dayrange,andamongthelowestat5days.Thelong‐termtrendintheperformanceofthewavemodelforecastsisalsoseenintheverificationagainstanalysis.Anomalycorrelationforsignificantwaveheighthasreachedsomeofitshighestvaluesin2014(Figure27).

ECMWFmaintainsaregularinter‐comparisonofperformancebetweenwavemodelsfromdifferentcentresonbehalfoftheExpertTeamonWavesandStormSurgesoftheWMO‐IOCJointTechnicalCommissionforOceanographyandMarineMeteorology(JCOMM).Thevariousforecastcentrescontributetothiscomparisonbyprovidingtheirforecastsatthelocationsoftheagreedsubsetofoceanbuoys(mainlylocatedinthenorthernhemisphere).AnexampleofthiscomparisonisshowninFigure28forthe12‐monthperiodJune2013–May2014.ECMWFforecastwindsareusedtodrivethewavemodelofMétéoFrance;thewavemodelsofthetwo



centresaresimilar,hencetheclosenessoftheirerrorsinFigure28.ECMWFoutperformstheothercentreswithregardtowindspeedandpeakperiod.Forwaveheight,MétéoFranceandECMWFforecastshavehighestskill.

AcomprehensivesetofwaveverificationchartsisavailableontheECMWFwebsiteat

http://www.ecmwf.int/en/forecasts/charts

under‘Oceanwaves’.

5. SevereweatherSupplementaryheadlinescoresforsevereweatherare:

TheskilloftheExtremeForecastIndex(EFI)for10mwindspeedverifiedusingtherelativeoperatingcharacteristicarea(Section5.1)

Thetropicalcyclonepositionerrorforthehigh‐resolutionforecast(Section5.2)

5.1. ExtremeForecastIndex(EFI)TheExtremeForecastIndex(EFI)wasdevelopedatECMWFasatooltoprovideearlywarningsforpotentiallyextremeevents.Bycomparingtheensembledistributionofachosenweatherparametertothemodel’sclimatologicaldistribution,theEFIindicatesoccasionswhenthereisanincreasedriskofanextremeeventoccurring.VerificationoftheEFIhasbeenperformedusingsynopticobservationsoverEuropefromtheGTS.Anextremeeventisjudgedtohaveoccurrediftheobservationexceedsthe95thpercentileoftheobservedclimateforthatstation(calculatedfroma15‐yearsample,1993–2007).TheabilityoftheEFItodetectextremeeventsisassessedusingtherelativeoperatingcharacteristic(ROC).Theheadlinemeasure,skilloftheEFIfor10mwindspeedatforecastday4(24‐hourperiod72–96hoursahead),isshowninFigure29(top),togetherwiththecorrespondingresultsfor24‐hourtotalprecipitation(centre)and2mtemperature(bottom).Eachcurveshowsseasonalvalues,aswellasthefour‐seasonrunningmean,ofROCareaskillscoresfrom2004to2014;thefinalpointoneachcurveincludesthespring(March–May)season2015.Forallthreeparameters,ROCskillhasstabilizedonahighlevel,withsomeinter‐annualvariationsduetoatmosphericvariability.

5.2. TropicalcyclonesThetropicalcyclonepositionerrorforthe3‐dayhigh‐resolutionforecastisoneofthetwosupplementaryheadlinescoresforsevereweather.Theaveragepositionerrorsforthehigh‐resolutionmedium‐rangeforecastsofalltropicalcyclones(alloceanbasins)overthelastten12‐monthperiodsareshowninFigure30.Errorsintheforecastintensityoftropicalcyclones,representedbythereportedsea‐levelpressureatthecentreofthesystem,arealsoshown.ThecomparisonofHRESandENScontroldemonstratesthebenefitofhigherresolutionfortropicalcycloneforecasts.

TheHRESandENSpositionerrors(topandbottompanels,Figure30)havereachedtheirlowestvaluessofar.ThesameistrueforthemeanabsolutespeederrorsoftheHRESandtheCTRLatD+3.Typicallytropicalcyclonesmovetooslowlyintheforecast,howeverthisnegativebiashasbeenrelativelysmallinrecentyears.Becauseofthesubstantialyear‐to‐yearvariationsinthenumberandintensityofcyclones,thereissomeuncertaintyinthesefigures.Boththemeanerror(bias)andmeanabsoluteerrorintropicalcycloneintensity(uppercentralpanelsin



Figure30)haveincreased.Aswiththespeederrors,thereisarelativelylargeuncertaintyinthesescoresbecauseoftheyear‐to‐yearvariationsinthenumberandcharacterofstorms.

ThebottompanelofFigure30showsthespreadanderrorofensembleforecastsoftropicalcycloneposition.Forreference,theHRESerrorisalsoshown.Theforecastwasunder‐dispersivebeforetheresolutionupgradein2010,butthespread‐errorrelationshiphasimprovedsincethen.ThefigurealsoshowsthattheHRESpositionerrorhasbeengenerallysmallerthantheensemblemeanerroratforecastday3(althoughverysimilarrecently),andviceversaatforecastday5.

TheensembletropicalcycloneforecastispresentedontheECMWFwebsiteasastrikeprobability:theprobabilityatanylocationthatareportedtropicalcyclonewillpasswithin120kmduringthenext120hours.Verificationoftheseprobabilisticforecastsforthethreelatest12‐monthperiodsisshowninFigure31.Resultsshowacertainamountofover‐confidence,withlittlechangefromyeartoyear.SkillisshownbytheROCandthemodifiedROC,thelatterusingthefalsealarmratio(fractionofyesforecaststhatturnouttobewrong)insteadofthefalsealarmrate(ratiooffalsealarmstothetotalnumberofnon‐events)onthehorizontalaxis.Thisremovesthereferencetonon‐eventsinthesampleandshowsmoreclearlythereductioninfalsealarmsinthosecaseswheretheeventisforecast.Differencesbetweenthelastthreeconsecutiveyearsofthesetwomeasuresarenotconsideredsignificant.

5.3. Additionalsevere‐weatherdiagnosticsWhilemanyscorestendtodegeneratetotrivialvaluesforrareevents,somehavebeenspecificallydesignedtoaddressthisissue.Hereweusethesymmetricextremaldependenceindex,SEDI(AnnexA.4),toevaluateheavyprecipitationforecastskilloftheHRES.Forecastsareverifiedagainstsynopticobservations.Figure32showsthetime‐evolutionofskillexpressedintermsofforecastdaysfor24‐hourprecipitationexceeding20mminEurope.Thegaininskillamountstoabouttwoforecastdaysoverthelast15yearsandisprimarilyduetoahigherhitrate.AmoredetailedevaluationofheavyprecipitationforecastskillcanbefoundinECMWFNewsletterNo.144.

6. Monthlyandseasonalforecasts

6.1. MonthlyforecastverificationstatisticsandperformanceThemonthlyforecastingsystemhasbeenintegratedwiththemedium‐rangeensemblesinceMarch2008.Thecombinedsystemmadeitpossibletoprovideuserswithensembleoutputuniformlyupto32daysahead,onceaweek.AsecondweeklyrunofthemonthlyforecastwasintroducedinOctober2011,runningeveryMonday(00UTC)toprovideanupdatetothemainThursdayrun.InIFScycle41r1(May2015)themonthlyensembleforecastsandre‐forecastshavebeenextendedfrom32to46days.

Figure33showstheprobabilisticperformanceofthemonthlyforecastovertheextratropicalnorthernhemisphereforsummer(JJA,toppanels)andwinter(DJF,bottompanels)seasonssinceSeptember2004forweek2(days12–18,leftpanels)andweek3+4(days19–32rightpanels).CurvesshowtheROCscorefortheprobabilitythatthe2mtemperatureisintheupperthirdoftheclimatedistributioninsummer,andinthelowerthirdoftheclimatedistributioninwinter.Thusitisameasureoftheabilityofthemodeltopredictwarmanomaliesinsummerandcoldanomaliesinwinter.Forreference,theROCscoreofthepersistenceforecastisalsoshownineachplot.Forecastskillforweek2exceedsthatofpersistencebyabout10%,for



weeks3to4(combined)byabout5%.Intheweeks3to4(14‐dayperiod),summerwarmanomaliesappeartohaveslightlyhigherpredictabilitythanwintercoldanomalies,althoughthelatterhasincreasedinrecentwinters(withtheexceptionof2012).Overall,theskilloftheforecastismorestablefromyeartoyearthantheskillofpersistence,bothinsummerandwinter.

ComprehensiveverificationforthemonthlyforecastsisavailableontheECMWFwebsiteat:

http://www.ecmwf.int/en/forecasts/charts

6.2. Seasonalforecastperformance

6.2.1. SeasonalforecastperformancefortheglobaldomainThecurrentversion(System4)oftheseasonalcomponentoftheIFSwasimplementedinNovember2011.ItusestheoceanmodelNEMOandECMWFatmosphericmodelcycle36r4.Theforecastscontain51ensemblemembersandthere‐forecasts15ensemblemembers,coveringaperiodof30years.

Asetofverificationstatisticsbasedonre‐forecastintegrations(1981–2010)fromSystem4hasbeenproducedandispresentedalongsidetheforecastproductsontheECMWFwebsite.

AcomprehensivedescriptionandassessmentofSystem4isprovidedinECMWFTechnicalMemorandum656,availablefromtheECMWFwebsite.

6.2.2. The2014–2015ElNiñoforecastsTheyear2014wascharacterizedbyachangefromslightlycoldtoslightlywarmconditionsintheeasterntropicalPacific.Themajorityofensemblemembersoftheforecastsmadeinspringandsummerof2014(uppertwopanelsintheleftcolumnofFigure34)predictedamoresubstantialstrengtheningofwarmanomaliesthanwhatwasobserved.Theautumnforecastcapturedthebasiccharacteristicsofthenextmonths’evolutionbetter,suggestinglittlechange.Thewinterforecast(bottomleftpanel)againpredictedastrongevolutiontowardsElNinoconditions,whichthistimeagreedverywellwithobservations.Themulti‐modelEUROSIPforecasts(rightcolumnofFigure34)performedslightlybetterinthefirsthalfoftheperiod,inthesensethattheensemblewasbettercentredontheobservations.TheFeb2015forecastofthestrongElNinodevelopmentin2015waspredictedmoreclearly(withgreatersharpness)intheECMWFmodel.

TheECMWFforecastsystempredictsthecontinuedstrengtheningoftheElNinointhecomingmonths,withthepeakexpectedaroundDecember2015.PreviousexperienceshowsthatforverylargeElNinoevents(specifically1997)themodeltendstoexaggeratetheamplitudeofSSTanomaliesduetonon‐linearitiesinthemodelbiascharacteristics.The2015ElNinoislikelytobeverystrong,butmorelikelythannotstillweakerthantheonein1997.Thelongerrangeforecast(13monthslead)suggeststhatElNinoislikelytoendsometimeinApril/May/June2016,withtemperatureseithernormalorbelownormalbyJuly2016.

6.2.3. TropicalstormpredictionsfromtheseasonalforecastsThe2014NorthAtlantichurricaneseasonwasquietwithanaccumulatedcycloneenergyindex(ACE)ofjust67%ofthe1950–2012climateaverage(seeFigure35).Thenumberoftropicalstormswhichformedin2014(8namedstorms)wasalsobelowaverage(12).SeasonaltropicalstormpredictionsfromSystem4indicatedbelowaverageactivitycomparedtoclimatologyover



theAtlantic.TheJuneforecastpredicted9(witharangefrom6to12)tropicalstormsintheAtlantic(Figure36)andanACEof60%oftheobservedclimatology(+/‐20%).Mostotherseasonalforecastmodelspredictedabelowaverage2014AtlantictropicalstormseasonduetothepresenceofamoderateEl‐Ninoevent.

Figure36showsthatSystem4predictedaboveaverageactivityovertheeasternNorthPacific(althoughwiththeACE10%belownormal)andslightlyenhancedactivityoverthewesternNorthPacific(ACE20%abovenormal).The2014easternPacifichurricaneseasonwaswellaboveaverage(19tropicalstormsformedovertheeasternNorthPacificbetweenJulyandDecember)andtheACEwas43%abovethe1981–2010average,whichmakesittheseventh‐highestsince1971.Only17tropicalstormsformedoverthewesternNorthPacificin2014betweenJulyandDecember,whichisbelowaverage(21.3).Therewasnoclearsignalintheforecastoverthisbasin.ThedropoftheACEintheAtlanticsectorwaswellcapturedbytheforecast,withalmostthesameamplitudeasobserved,althoughmostensemblememberspredictedatthetimeastrongerEl‐Nino(conducivetoareductionintropicalcycloneactivityintheAtlantic)thanobserved(Figure34).

6.2.4. Extratropicalseasonalforecasts2mtemperaturesinthenorthern‐hemispherewinter(DJF2014–15)werecharacterizedbystrongwarmanomaliesovernorthernEurasia,whichwerecapturedquitewellbytheseasonalforecast(Figure37).ThewesternpartsofEurope,incontrast,wereinfluencedbyapersistentcoldanomalyovertheNorthAtlantic.ThisanomalywasalsocapturedbythemodelbutitdidnotextendquiteasfarintoEuropeaswasobserved.Theseasonalforecastwasnotabletopredictthelarge‐scalecoldanomalyovermuchoftheeasternUnitedStatesandCanada.

Duringthenorthern‐hemispheresummer(JJA2015),centralandsouthernEuropeexperiencedextremelypersistentheat,leadingtoamagnitudeofseasonal2mtemperatureanomalyofmorethan2standarddeviationsabovenormal(Figure38).InsomeEuropeancountriesall‐timerecordtemperaturesweresurpassed.TemperaturesinnorthernEuropewereatthesametimebelownormal,withasteepgradientoftheanomalyatabout50‐55degreesnorth.TheseasonalforecastpredictedpositiveanomaliesinsouthernEurope,andnon‐significantanomaliesinnorthernEurope.TheextensionoftheanomalouswarmthintoAsiawaswellcaptured.



anomalycorrelation RMS error SEEPS

Europe

against analysis

Geopotential

100hPa ▴▲▴▴▴▴░░░░ ▲▲▲▲▲▴▴░░░ 500hPa ▴▴▴░░░░░░░ ▲▴▴▴▴░░░░░ 850hPa ▴▴▴░░░░░░░ ▴▲▴▴▴░░░░░ 1000hPa ▴▴▴░▴▴░░░░ ▲▲▴▴▴░░░░░

MSL pressure ░▴▴░░░░░░░ ▴▴▴░░░░░░░

Temperature

100hPa ▲▲▲▴▴▴░░░░ ▲▲▲▲▲▲▴▴░▴ 500hPa ▴▴▴▴░░░░░░ ▴▴▴▴▴▴░░░░ 850hPa ▲▴▴░▴▴░░░░ ▲▴▴░▴░░░░░ 1000hPa ▴▴▴▴▴▴░░░░ ▴▴▴▴▴▴▴░░░

Wind 200hPa ▲▴▴▴▴▴░░░░ ▲▲▴▴▴░░░░░ 850hPa ▴▴▴▴▴░░░░░ ▴▴▴▴▴░░░░░

Relative humidity

300hPa ▴░▴░░░░░░░ ▴▴▴░░░░░░░ 700hPa ▲▴▴▴▴▴░░░░ ▲▲▲▴▴▴░░▴░

against observations

Temperature

100hPa ░░░░░░░░░░ ▲▲▲▲▴▴▴░░░ 200hPa ░▴▴░░░░░░░ ░░▴░░░░░░░ 850hPa ▴░░░░░░░░░ ░░░░░░░░░░

2m temperature ▴▴▲▴▴▴▴░▴░

Wind

100hPa ▴▴▴░░░░░░░ ▴▴░░░░░░░░ 200hPa ░▴░░░░░░░░ ░░░░░░░░░░ 850hPa ░░░░░░░░░░ ░▴░░░░░░░░

10m wind

░░░░░░░░░░ 2m dew-point ▾░░▴▴░▴▴▴░ Total cloud cover ▲▲▲▴░▴▴░░░ 24h precipitation ▴▴▴░░░░░░░

Extratropical Northern Hemisphere

against analysis

Geopotential

100hPa ▲▲▲▲▲▴▴▴▴▴ ▲▲▲▲▲▲▲▲▲▴ 500hPa ▲▲▲▴▴▴▴░░░ ▲▲▲▴▴▴▴▴▴░ 850hPa ▲▲▲▴▴▴▴▴░░ ▲▲▲▴▴▴▴▴▴▴ 1000hPa ▲▲▴▴▴▴▴▴░░ ▲▲▴▴▴▴▴▴░░

MSL pressure ▴▴▴░░▴▴░░░ ▴▴▴░░▴▴░░░

Temperature

100hPa ▲▲▲▲▴▴▴▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 500hPa ▲▲▲▴▴▴▴▴░░ ▲▲▲▴▴▴▴▴▴▴ 850hPa ▲▴▴▴▴▴▴░░░ ▴░░░░░░░░░ 1000hPa ▴▴▴▲▴▴▴▴▴░ ▲▲▲▲▴▴▴▴▴▴

Wind 200hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▲▴▴▴▴▴▴ 850hPa ▲▴▴▴▴▴▴▴▴▴ ▲▴▴░▴▴▴▴▴▴

10m wind over ocean

▴▴░░░░▴░▴▴ ░░░░░░░░▴▴

Ocean wave height ▼▾▾░░▴▴░░░ ▼▾░░░░▴░▴░

Ocean wave period ▼▼▼▼▾░░░░░ ▼▼▼▼▾▾░░░░

Relative humidity

300hPa ▲▴▴▴▴▴░░░░ ▲▴▴▴▴▴▴▴▴▴ 700hPa ▲▴▴▴▴▴░░▴▴ ▲▲▲▲▲▲▲▲▲▲


Temperature

100hPa ▴▴▴▴░░░░░░ ▲▲▲▲▲▲▲▴▴▴ 200hPa ▴▴▴▴░░░░░░ ▴▴▴░░░░░░░ 850hPa ▾▾░░░░░░░░ ▾▾░░░░░░░░

2m temperature ▲▲▲▲▴▴▴▴▴░

Wind

100hPa ▲▲▴▴▴░░░░░ ▲▲▴▴▴▴░░░░ 200hPa ▲▴▴▴░░░░░░ ▲▴▴▴░░░░░░ 850hPa ▾░░░░░░░░░ ▾░░░░░░░░░

10m wind

▲▲▲▲▴▴░░░░ 2m dew-point ▴▲▲▲▲▲▴▴▴▴ Total cloud cover ▴▴▴▴▴▴▴▴▴▴ 24h precipitation ░░░░░░░░░░

Extratropical Southern Hemisphere

against analysis

Geopotential

100hPa ▴▴▴▴▴▴▴▴░░ ▲▲▲▲▲▲▲▲▲▲ 500hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴ 850hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴ 1000hPa ▲▲▲▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴

MSL pressure ▲▲▴▴▴▴▴▴▴▴ ▲▲▴▴▴▴▴▴▴▴

Temperature

100hPa ▲▲▲▲▲▲▴▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 500hPa ▲▲▲▴▴▴▴░▴▴ ▲▲▲▲▴▴▴▴▴▴ 850hPa ▲▴▴░░░░░░░ ░▾▾▾▾▾░▾▾▾ 1000hPa ▲▲▲▲▴▴▴▴▴░ ▲▲▴▴▴░░░░░



anomalycorrelation RMS error SEEPS

Wind 200hPa ▴▴▴▴▴▴▴▴░░ ▲▴▴▴▴▴▴▴▴▴ 850hPa ▲▲▴▴▴▴▴▴▴▴ ▲▲▲▴▴▴▴▴▴▴

10m wind over ocean

▲▴▴▴▴▴▴▴▴▴ ▴▴▴▴▴░░░░▴

Ocean wave height ▼▼░░░░░▴▴▴ ▼▼░░░░░▴░▴

Ocean wave period ▼▼▼▼▼▾░░░░ ▼▼▼▼▼▾▾▾░░

Relative humidity

300hPa ▴▴▴▴▴▴▴▴▴▴ ░░▴▴▴▴▴▴▴▴ 700hPa ▲▴▴▴▴░░░░░ ▲▲▲▲▲▲▲▲▲▲


Temperature

100hPa ▴▴▴▴▴▴▴▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 200hPa ▴▴▴▴▴▴▴▴▴░ ▴▴▴▴▴░▴▴▴░ 850hPa ░░░░░░▴░░░ ░░░░░░▴░░░

2m temperature ▴▴▴▴▴▴▴▴░░

Wind

100hPa ▲▲▲▴▴▴▴░░░ ▲▲▲▲▴▲▴▴▴▴ 200hPa ░▴▴▴▴▴▴▴░░ ░▴▴▴▴▴▴▴░▴ 850hPa ▴▴░▴░▴░░░░ ▴▴▴▴▴▴▴░░░

10m wind

▾▼▾▾▾░░░░░ 2m dew-point ░░▾░░░▴░░░ Total cloud cover ▲▴▴▴▴▴▴░▴░ 24h precipitation ░░░░░░▴░░░

Tropics

against analysis

Temperature

100hPa ▲▲▲▲▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲▲▲

500hPa ▲▲▲▲▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲▲▲

850hPa ▲▲▴▴▴▴▴░░░ ▲▴░░░░░░░░ 1000hPa ░▾▾░░░░░▾▾ ▲▲▲▲▲▲▲▲▲▴

Wind 200hPa ▲▲▲▲▲▲▲▲▲▲ ▲▲▲▲▲▲▲▲▲▲

850hPa ▲▲▲▲▲▲▲▴▴▴ ▲▲▲▴▴▴▴▴░░ 10m wind over ocean

▲▲▲▲▲▲▲▴▴▴ ▲▲▲▲▴▴░░▾░

Ocean wave height ▼▼▼▼▼▾▾▾▾▾ ▼▼▼▼▼▼▼▼▼▾

Ocean wave period ▼▼▼▼▼▾▾▾▾░ ▼▼▼▼▼▼▼▼▾▾

Relative humidity

300hPa ░▴▴▴▴▴▴▴▴▴ ▼▼▾▾▾▾▾▾▾▾ 700hPa ▲▴▴▴▴▴▴░░░ ▲▲▲▲▲▲▲▲▲▲


Temperature

100hPa ▲▲▲▲▲▲▲▲▲▴ ▲▲▲▲▲▲▲▲▲▲ 200hPa ▲▲▲▲▲▲▲▴▲▴ ▲▴░░░░░░░░ 850hPa ░░░░░░░░░░ ▴▴▴▴▴▴▴▴░░

2m temperature ▲▲▲▲▲▲▲▲▲▲

Wind

100hPa ▲▲▲▲▲▲▲▴▴▴ ▲▲▲▲▲▲▲▲▲▲ 200hPa ▴░▴▴▴▴▴░░░ ▴░▴▴▴▴▴▴▴▴ 850hPa ▲▲▲▴▴▴▴▴▴▴ ▴▴▴▴▴▴░░░░

10m wind

▼▼▼▼▼▼▼▼▼▾ 2m dew-point ▲▲▲▲▲▲▲▲▲▲

Total cloud cover ▲▲▲▲▲▲▲▲▲▲

24h precipitation ▴▴▴▴░░░░░░ Symbol legend: for a given forecast step... (d: score difference, s: confidence interval width) ▲ CY41r1 better than CY40r1 statistically highly significant (the confidence bar above zero by more than its height) (d/s>3) ▴ CY41r1 better than CY40r1 statistically significant (d/s≥1) ░ CY41r1 better than CY40r1, yet not statistically significant (d/s≥0.5) ░ not really any difference between CY40r1 and CY41r1 ░ CY41r1 worse than CY40r1, yet not statistically significant (d/s≤-0.5) ▾ CY41r1 worse than CY40r1 statistically significant (d/s≤-1) ▼ CY41r1 worse than CY40r1 statistically highly significant (the confidence bar below zero by more than its height) (d/s<-3)

Figure1: SummaryscorecardforCy41r1.Scorecardforcycle41r1versuscycle40r1verifiedbytherespectiveanalysesandobservationsat00and12UTCfor704forecastrunsintheperiod2January2014to10May2015.



Figure2:Primaryheadlinescore for thehigh‐resolution forecasts.Evolutionwith timeof the500hPageopotentialheight forecastperformance–eachpointon thecurves is the forecast rangeatwhich themonthlymean(bluelines)or12‐monthmeancentredonthatmonth(redline)oftheforecastanomalycorrelation (ACC) with the verifying analysis falls below 80% for Europe (top), northern hemisphereextratropics(centre)andsouthernhemisphereextratropics(bottom).

Europe

NH

SH



Figure3:Rootmeansquare(RMS)errorofforecastsof500hPageopotentialheight(m)atday6(red),verifiedagainstanalysis.Forcomparison,areferenceforecastmadebypersistingtheanalysisover6daysisshown(blue).Plottedvaluesare12‐monthmovingaverages;thelastpointonthecurvesisforthe12‐month periodAugust 2014–July 2015. Results are shown for the northern extra‐tropics (top), and thesouthernextra‐tropics(bottom).

NH

SH



Figure 4: Consistency of the 500 hPa height forecasts over Europe (top) and northern extratropics(bottom).CurvesshowthemonthlyaverageRMSdifferencebetweenforecastsforthesameverificationtimebutinitialised24hapart,for96–120h(blue)and120–144h(red).12‐monthmovingaveragescoresarealsoshown(inbold).

Europe

NH



Figure 5: Model scores for temperature (top) and wind (bottom) in the northern extratropicalstratosphere.CurvesshowthemonthlyaverageRMStemperatureandvectorwinderrorat50hPaforone‐day(blue)andfive‐day(red)forecasts,verifiedagainstanalysis.12‐monthmovingaveragescoresarealsoshown(inbold).

50hPatemperature

50hPawindspeed



Figure6:Normaliseddifferencesbetweenmodelcycles41r1and40r1ofthestandarddeviationofday5forecastdepartures(normalisedbytheobservationerror)ofGPSRObendingangles(upperleftpanel)andGPSROtemperatureretrievals(upperrightpanel)overthenorthernhemisphereextra‐tropics.Valuesbelow100indicateareductioninthestandarddeviationcomparedtothereference.Thebottompanelshowscorrespondingtimeseriesofthestandarddeviationof5‐dayforecastdeparturesofGPSRObendinganglesforthelayerbetween40and49km(blue:40r1,red:41r1).



Figure7:Primaryheadlinescorefortheensembleprobabilisticforecasts.Evolutionwithtimeof850hPatemperature ensemble forecast performance, verified against analysis. Each point on the curves is theforecastrangeatwhichthe3‐monthmean(bluelines)or12‐monthmeancentredonthatmonth(redline)of the continuous ranked probability skill score (CPRSS) falls below 25% for Europe (top), northernhemisphereextratropics(bottom).



Figure8:Ensemble spread (standarddeviation,dashed lines)andRMSerrorofensemble‐mean (solidlines) forwinter2014–2015(upper figure ineachpanel),anddifferencesofensemblespreadandRMSerrorofensemblemeanforlastthreewinterseasons(lowerfigureineachpanel,negativevaluesindicatespreadistoosmall);verificationisagainstanalysis,plotsarefor500hPageopotential(top)and850hPatemperature(bottom)overtheextratropicalnorthernhemisphereforforecastdays1to15.



Figure9:Ensemblespreadreliabilityofdifferentglobalmodelsfor500hPageopotentialinDJF2014–15inthenorthernhemisphereextra‐tropics(top)andinEurope(bottom)forday1(left)andday6(right),verifiedagainstanalysis.Circlesshowerrorfordifferentvaluesofspread,starsshowaverageerror‐spreadrelationship.



Figure10:Ensemblespreadreliabilityofdifferentglobalmodelsfor850hPatemperatureinDJF2014–15inthenorthernhemisphereextra‐tropics(top),Europe(centre),andthetropics(bottom)forday1(left)andday6(right),verifiedagainstanalysis.Circlesshowerrorfordifferentvaluesofspread,starsshowaverageerror‐spreadrelationship.



Figure11:CRPSfortemperatureat850hPainthenorthern(top)andsouthern(bottom)extratropicsatday5,verifiedagainstanalysis.Scoresareshownfortheensembleforecast(red)andthedressedERA‐Interimforecast(blue).BlackcurvesshowtheskilloftheENSrelativetothedressedERA‐Interimforecast.Valuesarerunning12‐monthaverages.NotethatforCRPS(redandbluecurves)lowervaluesarebetter,whileforCRPSskill(blackcurve)highervaluesarebetter.



Figure12:Forecastperformanceinthetropics.CurvesshowthemonthlyaverageRMSvectorwinderrorsat200hPa(top)and850hPa(bottom)forone‐day(blue)andfive‐day(red) forecasts,verifiedagainstanalysis.12‐monthmovingaveragescoresarealsoshown(inbold).

200hPa

850hPa



Figure13:WMO‐exchangedscoresfromglobalforecastcentres.RMSerrorof500hPageopotentialheightovernorthern(top)andsouthern(bottom)extratropics.Ineachpaneltheuppercurvesshowthesix‐dayforecasterrorandthelowercurvesshowthetwo‐dayforecasterror.Eachmodelisverifiedagainstitsownanalysis.JMA=JapanMeteorologicalAgency,CMC=CanadianMeteorologicalCentre,UKMO=theUKMetOffice,NCEP=U.S.NationalCentersforEnvironmentalPrediction,M‐F=MétéoFrance.



Figure14:WMO‐exchangedscoresforverificationagainstradiosondes:500hPaheight(top)and850hPawind(bottom)RMSerroroverEurope(annualmeanAugust2014–July2015).



Figure15:AsFigure14forthenorthernhemisphereextratropics.



Figure16:WMO‐exchangedscores fromglobal forecastcentres.RMSvectorwinderrorovertropicsat250hPa(top)and850hPa(bottom).Ineachpaneltheuppercurvesshowthefive‐dayforecasterrorandthelowercurvesshowtheone‐dayforecasterror.Eachmodelisverifiedagainstitsownanalysis.



Figure17:AsFigure16forverificationagainstradiosondeobservations.



Figure18:Supplementaryheadlinescoresfordeterministic(top,centre)andprobabilistic(bottom)precipitationforecasts.Theevaluationisfor24‐hourtotalprecipitationverifiedagainstsynopticobservationsintheextratropics;eachpointiscalculatedovera12‐monthperiod,plottedatthecentreoftheperiod.ThedashedcurveshowsthedeterministicheadlinescoreforERA‐Interimasareference.ThecentrepanelshowsthedifferencebetweentheoperationalforecastandERA‐Interim.Curvesshowthenumberofdaysforwhichthecentred12‐monthmeanskillremainsaboveaspecifiedthreshold.Theforecastdayonthey‐axisistheendofthe24‐hourperiodoverwhichtheprecipitationisaccumulated.

SEEPS

ERA‐Interim

SEEPSdifference

HRES

HRES‐ERA‐Interim

CRPSS



Figure19:ComparisonofprecipitationforecastskillforECMWF(red),theMetOffice(UKMO,blue),JapanMeteorologicalAgency (JMA,magenta) andNCEP (green) using the supplementaryheadline scores forprecipitation shown in Figure 18. Top: deterministic; bottom: probabilistic skill. Curves show the skillcomputedoverallavailablesynopticstationsintheextratropicsforforecastsfromAugust2014–July2015.Barsindicate95%confidenceintervals.

HRESprecipitation‐SEEPS

ENSprecipitation‐CRPSS



Figure20:Verificationof2mtemperatureforecastsagainstEuropeanSYNOPdataontheGTSfor60‐hour(night‐time)and72‐hour(daytime)forecasts.Lowerpairofcurvesshowsbias,uppercurvesarestandarddeviationoferror.

Figure 21: Verification of 2m dewpoint forecasts against European SYNOP data on the GlobalTelecommunicationSystem(GTS)for60‐hour(night‐time)and72‐hour(daytime)forecasts.Lowerpairofcurvesshowsbias,uppercurvesshowstandarddeviationoferror.

2‐mtemperature

2‐mdewpoint



Figure22:VerificationoftotalcloudcoverforecastsagainstEuropeanSYNOPdataontheGTSfor60‐hour(night‐time) and 72‐hour (daytime) forecasts. Lower pair of curves shows bias, upper curves showstandarddeviationoferror.

Figure23:Verificationof10mwindspeedforecastsagainstEuropeanSYNOPdataontheGTSfor60‐hour(night‐time) and 72‐hour (daytime) forecasts. Lower pair of curves shows bias, upper curves showstandarddeviationoferror.

Totalcloudcover

10‐mwindspeed



Figure24:12‐monthrunningaverageoftheday3forecastskillrelativetoERA‐InterimofnormalizedTOAreflectedsolarflux(dailytotals),verifiedagainstsatellitedata.Theverificationhasbeencarriedoutforthose parts of the northern hemisphere extratropics (green), tropics (red), and southern hemisphereextratropics(blue)whicharecoveredbytheCM‐SAFproduct(approximately70Sto70N,and70Wto70E).



Figure25:EvolutionofskilloftheHRESforecastatday5,expressedasrelativeskillcomparedtoERA‐Interim.Verificationisagainstanalysisfor500hPageopotential(Z500),850hPatemperature(T850),andmeansealevelpressure(MSLP),usingRMSEasametric.VerificationisagainstSYNOPfor2mtemperature(T2M),10mwindspeed(V10),andtotalcloudcover(TCC),usingerrorstandarddeviationasametric.



Figure26: Time series of verification of the ECMWF10mwind forecast (top panel) andwavemodelforecast(waveheight,bottompanel)verifiedagainstnorthernhemispherebuoyobservations.Thescatterindexistheerrorstandarddeviationnormalisedbythemeanobservedvalue;athree‐monthrunningmeanisused.



Figure27:Oceanwaveforecasts.Monthlyscoreand12‐monthrunningmean(bold)ofACCforoceanwaveheightsverifiedagainstanalysisforthenorthern(top)andsouthernextratropics(bottom)atday1(blue),5(red)and10(green).



Figure28:Verificationofdifferentmodelforecastsofwaveheight,10mwindspeedandpeakwaveperiodusingaconsistentsetofobservationsfromwavebuoys.Thescatterindex(SI)isthestandarddeviationoferrornormalisedbythemeanobservedvalue;plotsshowtheSIforthe12‐monthperiodJuly2014–June2015.Thex‐axisshowstheforecastrangeindaysfromanalysis(step0)today5.MOF:MetOffice,UK;FNM:FleetNumericalMeteorologyandOceanographyCentre,USA;NCP:NationalCenters forEnvironmentalPrediction, USA; MTF: Météo‐France; DWD: Deutscher Wetterdienst, BoM: Bureau of Meteorology,Australia;JMA:JapanMeteorologicalAgency;KMA:KoreaMeteorologicalAdministration.



Figure 29: Verification of Extreme Forecast Index (EFI) against analysis. Top panel: supplementaryheadlinescore–skilloftheEFIfor10mwindspeedatforecastday4(24‐hourperiod72–96hoursahead);anextremeevent is takenasanobservationexceeding95thpercentileof stationclimate.Curvesshowseasonalvalues(dotted)andfour‐seasonrunningmean(continuous)ofrelativeoperatingcharacteristic(ROC) area skill scores. Centre and bottom panels show the equivalent ROC area skill scores forprecipitationEFIforecastsandfor2mtemperatureEFIforecasts.

2‐mtemperature

10‐mwindspeed

24‐hprecipitation



Figure30:Verificationoftropicalcyclonepredictionsfromtheoperationalhigh‐resolutionandensembleforecast. Results are shown for all tropical cyclones occurring globally in 12‐month periods ending on30June. Verification is against the observed position reported via the GTS. Top panel supplementaryheadlinescore–themeanpositionerror(km)ofthethree‐dayhigh‐resolutionforecast.Theerrorforday5isincludedforcomparison.Centrefourpanelsshowmeanerror(bias)inthecycloneintensity(differencebetweenforecastandreportedcentralpressure;positiveerrorindicatestheforecastpressureislessdeepthanobserved),meanabsoluteerroroftheintensityandmeanandabsoluteerrorofcyclonemotionspeedforcycloneforecastbothbyHRESandENScontrol.Bottompanelshowsmeanpositionerrorofensemblemean(meanofcyclonesforecastbyensemblemembers)withrespecttotheobservedcyclone(cyancurve)andensemblespread(meanofdistancesofensemblecyclonesfromtheensemblemean;redcurve);forcomparisontheHRESpositionerror(fromthetoppanel)isplottedaswell(bluecurve).



Figure31:Probabilisticverificationofensembletropicalcycloneforecastsatday10forthree12‐monthperiods:July2012–June2013(green),July2013–June2014(blue)andJuly2014–June2015(red).Upperpanelshowsreliabilitydiagram(theclosertothediagonal,thebetter).Thelowerpanelshows(left)thestandardROCdiagramand(right)amodifiedROCdiagram,wherethefalsealarmratioisusedinsteadofthefalsealarmrate.ForbothROCandmodifiedROC,thecloserthecurveistotheupper‐leftcorner,thebetter,indicatingagreaterproportionofhits,andfewerfalsealarms.



Figure32:EvolutionofskilloftheHRESforecastinpredicting24‐hprecipitationamounts>20mmintheextra‐tropicsasmeasuredbytheSEDIscore,expressedintermsofforecastdays.VerificationisagainstSYNOPobservations.NumbersontherightindicatedifferentSEDIthresholdsused.Curvesshow12‐monthrunningaverages.



Figure 33: Verification of the monthly forecast against analysis. Area under the ROC curve for theprobabilitythat2mtemperatureisintheupperthirdoftheclimatedistributioninsummer(top)andinthelowerthirdinwinter(bottom).Scoresarecalculatedforeachthree‐monthseasonforalllandpointsintheextra‐tropicalnorthernhemisphere.Leftpanelsshowthescoreoftheoperationalmonthlyforecastingsystemforforecastdays12–18(7‐daymean),andrightpanelsforforecastdays19–32(14‐daymean).Asa reference, lighter coloured lines shows the scoreusingpersistence of thepreceding7‐day or 14‐dayperiodoftheforecast.



Figure34:ECMWF(leftcolumn)andEUROSIPmulti‐modelforecast(rightcolumn)seasonalforecastsofSSTanomaliesovertheNINO3.4regionofthetropicalPacificfrom(toptobottomrows)May2014,August2014,November2014andFebruary2015.Theredlinesrepresenttheensemblemembers;dottedbluelineshowsthesubsequentverification.



Figure35:Timeseriesofaccumulatedcycloneenergy(ACE)fortheAtlantictropicalstormseasonsJuly–December1990toJuly–December2014.Bluelineindicatestheensemblemeanforecastsandgreenbarsshowtheassociateduncertainty(±1standarddeviation);reddottedlineshowsobservations.ForecastsarefromSystem4oftheseasonalcomponentoftheIFS:thesearebasedonthe15‐memberre‐forecasts;from2011onwardstheyarefromtheoperational51‐memberseasonalforecastensemble.Startdateoftheforecastis1June.



Figure36:TropicalstormfrequencyforecastissuedinJune2014forthesix‐monthperiodJuly–December2014.Greenbarsrepresenttheforecastnumberoftropicalstormsineachoceanbasin(ensemblemean);orangebarsrepresentclimatology.Thevaluesofeachbararewritteninblackunderneath.Theblackbarsrepresent±1standarddeviationwithintheensembledistribution;thesevaluesareindicatedbythebluenumber.The51‐memberensembleforecastiscomparedwiththeclimatology.AWilcoxon‐Mann‐Whitney(WMW)testisthenappliedtoevaluateifthepredictedtropicalstormfrequenciesaresignificantlydifferentfromtheclimatology.TheoceanbasinswheretheWMWtestdetectssignificancelargerthan90%haveashadedbackground.



Figure37:Anomalyof2mtemperatureaspredictedbytheseasonalforecastfromNovember2014forDJF2014/15(upperpanel),andverifyinganalysis(lowerpanel).Blackcontoursintheanalysisindicateregionswhereanomaliesexceed1.5standarddeviations.

Evaluationof ECMWFforecastsincluding2014‐2015upgrades


Figure38:Anomalyof2mtemperatureaspredictedbytheseasonalforecastfromMay2015forJJA2015(upperpanel),andverifyinganalysis(lowerpanel).Blackcontoursintheanalysisindicateregionswhereanomaliesexceed1.5standarddeviations.



Ashortnoteonscoresusedinthisreport

A.1 Deterministicupper‐airforecastsTheverificationsusedfollowWMOCBSrecommendationsascloselyaspossible.Scoresarecomputedfromforecastsonastandard1.5×1.5grid(computedfromspectralfieldswithT120truncation)limitedtostandarddomains(boundingco‐ordinatesarereproducedinthefigureinnercaptions),asthisistheresolutionagreedintheupdatedWMOCBSrecommendationsapprovedbythe16thWMOCongressin2011.Whenothercentres’scoresareproduced,theyhavebeenprovidedaspartoftheWMOCBSexchangeofscoresamongGDPScentres,unlessstatedotherwise–e.g.whenverificationscoresarecomputedusingradiosondedata(Figure14),thesondeshavebeenselectedfollowinganagreementreachedbydatamonitoringcentresandpublishedintheWMOWWWOperationalNewsletter.

Rootmeansquareerrors(RMSE)arethesquarerootofthegeographicalaverageofthesquareddifferencesbetweentheforecastfieldandtheanalysisvalidforthesametime.Whenmodelsarecompared,eachmodelusesitsownanalysisforverification;RMSEforwinds(Figure14,Figure16)arecomputedbytakingtherootofthesumsofthemeansquarederrorsforthetwocomponentsofthewindindependently.

SkillscoresarecomputedasthereductioninRMSEachievedbythemodelwithrespecttopersistence(forecastobtainedbypersistingtheinitialanalysisovertheforecastrange);inmathematicalterms:

2

2

1*100p

f

RMSE

RMSESS

Figure2showscorrelationsinspacebetweentheforecastanomalyandtheverifyinganalysisanomaly.AnomalieswithrespecttoERA‐InterimanalysisclimateareavailableatECMWFfromearly1980s.Foroceanwaves(Figure27)theclimatehasbeenalsoderivedfromtheERA‐Interimanalyses.

A.2 ProbabilisticforecastsEventsfortheverificationofmedium‐rangeprobabilisticforecastsareusuallydefinedasanomalieswithreferencetoasuitableclimatology.Forupper‐airparameters,theclimateisderivedfromERA‐Interimanalysesforthe20‐yearperiod1989–2008.Probabilisticskillisevaluatedinthisreportusingthecontinuousrankedprobabilityskillscore(CRPSS)andtheareaunderrelativeoperatingcharacteristic(ROC)curve.

Thecontinuousrankedprobabilityscore(CRPS),anintegralmeasureofthequalityoftheforecastprobabilitydistribution,iscomputedas

where isforecastprobabilitycumulativedistributionfunction(CDF)and isanalysedvalue

expressedasaCDF.CRPSiscomputeddiscretelyfollowingHersbach,2000.CRPSSisthencomputedas

Evaluationof ECMWFforecastsincluding2014‐2015upgrades


1

whereCRPSclimistheCRPSofaclimateforecast(basedeitherontheERA‐Interimanalysisorobservedclimatology).CRPSSisusedtomeasurethelong‐termevolutionofskilloftheIFSensemble(Figure7)anditsinter‐annualvariability(Figure11).

ROCcurvesshowhowmuchsignalcanbegainedfromtheensembleforecast.Althoughasinglevaluedforecastcanbecharacterisedbyauniquefalsealarm(x‐axis)andhitrate(y‐axis),ensembleforecastscanbeusedtodetectthesignalindifferentways,dependingonwhethertheforecastuserismoresensitivetothenumberofhits(theforecastwillbeissued,evenifarelativelysmallnumberofmembersforecasttheevent)oroffalsealarms(onewillthenwaitforalargeproportionofmemberstoforecasttheevent).TheROCcurvesimplyshowsthefalsealarmandhitratesassociatedwiththedifferentthresholds(proportionofmembersorprobabilities)used,beforetheforecastisissued(Figure31).Figure31alsoshowsamodifiedROCplotofhitrateagainstfalsealarmratio(fractionofyesforecaststhatturnouttobewrong)insteadofthefalsealarmrate(ratiooffalsealarmstothetotalnumberofnon‐events).

Sincetheclosertotheupperleftcorner(0falsealarm,100%hits)thebetter,theareaundertheROCcurve(ROCA)isagoodindicationoftheforecastskill(0.5isnoskill,1isperfectdetection).TimeseriesoftheROCAareshowninFigure33.

A. 3 Weather parameters (Section 4) VerificationofthedeterministicprecipitationforecastsismadeusingthenewlydevelopedSEEPSscore(Rodwelletal.,2010).SEEPS(stableequitableerrorinprobabilityspace)usesthreecategories:dry,lightprecipitation,andheavyprecipitation.Here“dry”isdefined,withreferencetoWMOguidelinesforobservationreporting,tobeanyaccumulation(roundedtothenearest0.1mm)thatislessthanorequalto0.2mm.Toensurethatthescoreisapplicableforanyclimaticregion,the“light”and“heavy”categoriesaredefinedbythelocalclimatologysothatlightprecipitationoccurstwiceasoftenasheavyprecipitation.Aglobal30‐yearclimatologyofSYNOPstationobservationsisused(theresultingthresholdbetweenthelightandheavycategoriesisgenerallybetween3and15mmforEurope,dependingonlocationandmonth).SEEPSisusedtocompare24‐houraccumulationsderivedfromglobalSYNOPobservations(exchangedovertheGlobalTelecommunicationSystem;GTS)withvaluesatthenearestmodelgrid‐point.1‐SEEPSisusedforpresentationalpurposes(Figure18,Figure19)asthisprovidesapositivelyorientedskillscore.

TheensembleprecipitationforecastsareevaluatedwiththeCRPSS(Figure18,Figure19).VerificationisagainstthesamesetofSYNOPobservationsasusedforthedeterministicforecast.

Forotherweatherparameters(Figure20toFigure23),verificationdataareEuropean6‐hourlySYNOPdata(areaboundariesarereportedaspartofthefigurecaptions).Modeldataareinterpolatedtostationlocationsusingbi‐linearinterpolationofthefourclosestgridpoints,providedthedifferencebetweenthemodelandtrueorographyislessthan500m.AcrudequalitycontrolisappliedtoSYNOPdata(maximumdeparturefromthemodelforecasthastobelessthan25K,20g/kgor15m/sfortemperature,specifichumidityandwindspeedrespectively).2mtemperaturesarecorrectedfordifferencesbetweenmodelandtrue



orography,usingacrudeconstantlapserateassumptionprovidedthecorrectionislessthan4Kamplitude(dataareotherwiserejected).

A.4 VerificationofrareeventsExperimentalverificationofdeterministicforecastsofrareeventsisperformedusingthesymmetricextremaldependenceindexSEDI(Figure32),whichiscomputedas

log log log 1 log 1log log log 1 log 1

whereFisthefalsealarmrateandHisthehitrate.InordertoobtainafaircomparisonbetweentwoforecastingsystemsusingSEDI,theforecastsneedtobecalibrated(FerroandStephenson,2011).ThereforeSEDIisameasureofthepotentialskillofaforecastsystem.Inordertogetafullerpictureoftheactualskill,thefrequencybiasoftheuncalibratedforecastcanbeanalysed.

ReferencesFerro,C.A.T.,andD.B.Stephenson,2011:Extremaldependenceindices:improvedverificationmeasuresfordeterministicforecastsofrarebinaryevents.Wea.Forecasting,26,699–713.

Hersbach,H., 2000:Decomposition of the Continuous Ranked Probability Score for EnsemblePredictionSystem.Wea.Forecasting,15,559–570.

Richardson,D. S., 2000: Skill and relative economic value of the ECMWFensemble predictionsystem.Q.J.R.Meteorol.Soc.,126,649–667.

Rodwell,M.J.,D.S.Richardson,T.D.Hewson,andT.Haiden,2010:Anewequitablescoresuitableforverifyingprecipitationinnumericalweatherprediction.Q.J.R.Meteorol.Soc.,136,1344–1363.

ecmwf - t. haiden, m. janousek, p. bauer, j. bidlot, m. dahoui, l. … · 2015. 12. 14. ·...

Documents