Slide 1

14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic 14-16 May 2001 Barbara G. Brown NCAR Boulder, Colorado, U.S.A. bgb@ucar.edu Slide 2 14 May 2001QPF Verification Workshop Why probability forecasts? the widespread practice of ignoring uncertainty when formulating and communicating forecasts represents an extreme form of inconsistency and generally results in the largest possible reductions in quality and value. --Murphy (1993) Slide 3 14 May 2001QPF Verification Workshop Outline 1.Background and basics Types of events Types of forecasts Representation of probabilistic forecasts in the verification framework Slide 4 14 May 2001QPF Verification Workshop Outline continued 2.Verification approaches: focus on 2-category case Measures Graphical representations Using statistical models Signal detection theory Ensemble forecast verification Extensions to multi-category verification problem Comparing probabilistic and categorical forecasts 3.Connections to value 4.Summary, conclusions, issues Slide 5 14 May 2001QPF Verification Workshop Background and basics Types of events: Two-category Multi-category Two-category events: Either event A happens or Event B happens Examples: Rain/No-rain Hail/No-hail Tornado/No-tornado Multi-category event Event A, B, C, .or Z happens Example: Precipitation categories (< 1 mm, 1-5 mm, 5-10 mm, etc.) Slide 6 14 May 2001QPF Verification Workshop Background and basics cont. Types of forecasts Completely confident Forecast probability is either 0 or 1 Example: Rain/No rain Probabilistic Objective (deterministic, statistical, ensemble-based) Subjective Probability is stated explicitly Slide 7 14 May 2001QPF Verification Workshop Background and basics cont. Representation of probabilistic forecasts in the verification framework x = 0 or 1 f = 0, , 1.0 f may be limited to only certain values between 0 and 1 Joint distribution: p(f,x), where x = 0, 1 Ex: If there are 12 possible values of f, then p(f,x) is comprised of 24 elements Slide 8 14 May 2001QPF Verification Workshop Background and basics, cont. Factorizations: Conditional and marginal probabilities Calibration-Refinement factorization: p(f,x) = p(x|f) p(f) p(x=0|f) = 1 p(x=1|f) = 1 E(x|f) Only one number is needed to specify the distribution p(x|f) for each f p(f) is the frequency of use of each forecast probability Likelihood-Base Rate factorization: p(f,x) = p(f|x) p(x) p(x) is the relative frequency of a Yes observation (e.g., the sample climatology of precipitation); p(x) = E(x) Slide 9 14 May 2001QPF Verification Workshop Attributes [from Murphy and Winkler(1992)] (sharpness) Slide 10 14 May 2001QPF Verification Workshop Use the counts in this table to compute various common statistics (e.g., POD, POFD, H-K, FAR, CSI, Bias, etc.) Verification approaches: 2x2 case Completely confident forecasts: Slide 11 14 May 2001QPF Verification Workshop Verification measures for 2x2 (Yes/No) completely confident forecasts Slide 12 14 May 2001QPF Verification Workshop Relationships among measures in the 2x2 case Many of the measures in the 2x2 case are strongly related in surprisingly complex ways. For example: Slide 13 14 May 2001QPF Verification Workshop The lines indicate different values of POD and POFD (where POD = POFD). From Brown and Young (2000) 0.10 0.30 0.50 0.70 0.90 Slide 14 14 May 2001QPF Verification Workshop CSI as a function of p(x=1) and POD=POFD 0.1 0.3 0.5 0.7 0.9 Slide 15 14 May 2001QPF Verification Workshop CSI as a function of FAR and POD Slide 16 14 May 2001QPF Verification Workshop Measures for Probabilistic Forecasts Summary measures: Expectation Conditional: E(f|x=0), E(f|x=1) E(x|f) Marginal: E(f) E(x) = p(x=1) Correlation Joint distribution Variability Conditional: Var.(f|x=0), Var(f|x=1) Var(x|f) Marginal : Var(f) Var(x) = E(x)[1-E(x)] Slide 17 14 May 2001QPF Verification Workshop From Murphy and Winkler (1992) Summary measures for joint and marginal distributions: Slide 18 14 May 2001QPF Verification Workshop From Murphy and Winkler (1992) Summary measures for conditional distributions: Slide 19 14 May 2001QPF Verification Workshop Performance measures Brier score: Analogous to MSE; negative orientation; For perfect forecasts: BS=0 Brier skill score: Analogous to MSE skill score Slide 20 14 May 2001QPF Verification Workshop From Murphy and Winkler (1992): Slide 21 14 May 2001QPF Verification Workshop Brier score displays From Shirey and Erickson, http://www.nws.noaa.gov/tdl/synop/amspapers/masmrfpap.htm Slide 22 14 May 2001QPF Verification Workshop Brier score displays From http://www.nws.noaa.gov/tdl/synop/mrfpop/mainframes.htm Slide 23 14 May 2001QPF Verification Workshop Decomposition of the Brier Score Break Brier score into more elemental components: ReliabilityResolutionUncertainty Where I = the number of distinct probability values and Then, the Brier Skill Score can be re-formulated as Slide 24 14 May 2001QPF Verification Workshop Graphical representations of measures Reliability diagram p(x=1|f i ) vs. f i Sharpness diagram p(f) Attributes diagram Reliability, Resolution, Skill/No-skill Discrimination diagram p(f|x=0) and p(f|x=1) Together, these diagrams provide a relatively complete picture of the quality of a set of probability forecasts Slide 25 14 May 2001QPF Verification Workshop Reliability and Sharpness (from Wilks 1995) ClimatologyMinimal RESUnderforecasting Good RES, at expense of REL Reliable forecasts of rare event Small sample size Slide 26 14 May 2001QPF Verification Workshop Reliability and Sharpness (from Murphy and Winkler 1992) St. Louis 12-24 h PoP Cool Season No skill No RES Model Sub Model Sub Slide 27 14 May 2001QPF Verification Workshop Attributes diagram (from Wilks 1995) Slide 28 14 May 2001QPF Verification Workshop Icing forecast examples Slide 29 14 May 2001QPF Verification Workshop Use of statistical models to describe verification features Exploratory study by Murphy and Wilks (1998) Case study Use regression model to model reliability Use Beta distribution to model p(f) as measure of sharpness Use multivariate diagram to display combinations of characteristics Promising approach that is worthy of more investigation Slide 30 14 May 2001QPF Verification Workshop Fit Beta distribution to p(f) 2 parameters: p. q 0 1 Ideal: p