Barbara Brown1, Tara Jensen1, Michelle Harrold1, Tressa Fowler1, Randy Bullock1, Eric Gilleland1, and Brian Etherton2

1NCAR/RAL - Joint Numerical Testbed Program2NOAA/ESRL - Global Systems Division

AndDevelopmental Testbed Center

Verification Methods for High Resolution Ensemble Forecasts

Warn-On-Forecast Workshop: 8-9 February 2012

Evaluation and Diagnostic

Large variability in space and time Difficult to identify meaningful

impactsExtreme, high impact, weather

eventsSmall regions of importance

Difficult to identify impacts of forecast “improvements” across whole domain

Verification scores Desire for a single score... But CSI

alone does not give a lot of information about performance or improvements

Relationships and dependencies among scores

Double penalty for displaced high-res forecasts

Challenges for objective evaluation of convective scale short-term prediction...

•ObservationsUncertainties in qualityNeed to be on time and spatial scaled that support evaluation

Traditional Verification Scores

CSI is a nonlinear function of POD and FARCSI depends on base rate (event frequency) and Bias

Ex: Relationships among scores

FAR

POD

CSI

Very different combinations of FAR and POD lead to the same CSI value

1CSI 1 1 1POD 1 FAR

PODBias1 FAR

Freq Bias

Results from HMT-West 2010-2011 seasonCourtesy of Ed Tollerud, DTC/HMT Collab

9km - Ensemble Mean – 6h Precip

6HR>0.1 in.

6HR>0.5 in.

6HR>1.0 in.

6HR>2.0 in.

Performance DiagramAll on same plotPOD1-FAR

(aka Success Ratio)CSIFreq Bias

Best

Success Ratio (1-FAR)

Freq

Bia

s

CSI0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Here we see:Decreasing skill with higher thresholds even with multiple metricsRoebber (WAF, 2009)

Wilson (presentation, 2008)

Dots representDifferent leads

Looking at variability in space and time

14 May 2009 Init: 00 UTC MODE Objects Thresh: 30dBZ

No RadarAssim.

Objects ForecastField

ObservedField

RadarAssim.

SolidFCSTOBJ

LineOBS OBJ

What if you had this: Instead of this:

Objects are 30% too big (MODE area ratio=1.3) FBIAS = 1.3 for grid

Shifted west 40 km(MODE centroid distance = 10 gs) POD = 0.35

Moving too slow(MODE-TD angle diff = 15%)

FAR = 0.72

Peak Rain 1/2” too much(MODE diff in 90th percentile of intensities = 0.5”)

CSI = 0.16

Comparing objects can give more descriptive assessment of the forecast

QPE field Probability Field

Traditional MetricsBrier Score: 0.07Area Under ROC: 0.62

Spatial MetricsCentroid Distance:Obj1) 200 kmObj2) 88kmArea Ratio:Obj1) 0.69Obj2) 0.65

1

2 Median Of MaxInterest: 0.77

Obj PODY: 0.72Obj FAR: 0.32

Okay Forecast with

Displacement Error?

Or Bad Forecastbecause Too

SharpOr

Underdispersive?

QPE_06 >1 in. 50% Prob (APCP_06> 1 in.)

Perfe

ct Reli

abilit

y

OverFcst

UnderFcst

Reflectivity > 30 dBZ MODE Objects (01-06hr)

Looks like there is definitely a timing error

What would a spaghetti plot of MODE objects look like?

Hatched - Observed OBJ Solid - Fcst OBJ

Centroid of object

MODE Time Domain - (MODE – TD)

West

East

Color gives W-Emovement

Tim

e in

crea

sing

Preparing for implementation in Model Evaluation Tools (MET) in next release (Spring 2012)

Applied to model forecasts from 2010 HWT Spring Experiment

Example Attribute of MODE-TD

Conv. Rad. = 10 grid squaresConv Thresh. = APCP_01 > 3 mm

HWT SE 2010 DataMay 17 – June 16, 2010 (18 forecasts/days)

Number of objects:Obs 143, ARPS 161ARW 172, NMM 238

E-W speed (m/s)

Duration (h)

90th percentile intensity

MODE-TD Attributes

Continue with traditional scores (and methods of display) and not gain a true sense of why the ensembles are performing well and poorly- Or -

Adopt performance diagrams, reliability diagrams, rank histograms, and spatial methods for diagnostics and understanding

Take aways: To evaluate high resolution ensembles, you could…

Model Evaluation Tools (MET) developed by the Developmental Testbed Center (DTC)

R statistics package, including R-spatial developed by Eric Gilleland at NCAR/RAL Joint Numerical Testbed (JNT) and R-verification

Where to get these methods?

Recommended websites:http://www.dtcenter.org/met/users

http://www.r-project.org/ http://www.cawcr.gov.au/projects/verification/

http://www.ral.ucar.edu/projects/icp/http://www.dtcenter.org/eval/hwt/2011

The DTC and JNT Program can also serve as a resource for Evaluation Techniques

Extra MaterialToolsIntercomparisonsOutreachCollaboration

www.dtcenter.org/met/usersMETv3.1AvailableNow

METv4.0With GRIB2 support and MODE-TD to be released in late spring 2012

Object Oriented Method: MODEHow it works

OBSENS FCSTRadius=5Thresh>0.25”

Radius=5Thresh>0.25””

Merging

Matching

No false alarms

Misses

Merging

Matched Object 1

Matched Object 2

Unmatched Object

1. Smooth Field 2. Threshold 3. Restore Intensities

4. Merge within 5. Match between 6. Calculate Attributesand Categorical Stats

Example from HMT-West 2010-2011 seasonCourtesy of DTC/HMT Collaboration

Volume / intersection / union / symmetric difference

Axis anglesSpatial orientationAverage speedCentriods – space and time

No way to combine calculation of centriod of space and time because of their differing units

Fuzzy logic handles them by weighting each independently

Metrics to be available in MODE-TD

“Completely updated chapter on the Verification of Spatial Forecasts taking account of the wealth of new research in the area “

Authors: Brown, Ebert, Gilleland

Jolliffe and Stephenson, 2nd Ed

Spatial verification methods in “R”

Implemented by Eric Gilleland

Includes all major spatial methods

R spatial verification package

Example: Neighborhood methods

ICPInternational effort

focused on comparison of capabilities of the different spatial verification methods

Central U.S. precipitation forecasts and observations from HWT 2005

Many publications in WAF special collection

Preparing now for ICP2

Intercomparison project (ICP) and other collaborations

Web site: http://www.ral.ucar.edu/projects/icp/

Other collaborations• Collaborations with

WMO/WWRP Working Groups on Mesoscale and Nowcasting Research– Documents on verification of

mesoscale forecasts and cloud forecasts

• Workshops and tutorials

International collaborationGoals

Further testing of spatial and other methods from ICP1

Application to complex terrainVariables: Precipitation, WindForecasts (Model output), Obs analyses and

observations from 2007 COPS and MAP / D-PhaseVERA analyses (Include ensemble observation

analyses)2 or more sets of model output from MAP D-

PHASE

ICP2