Exeter, February 2008Exeter, February 2008 11

The impenetrable hedge: The impenetrable hedge: some thoughts on propriety, some thoughts on propriety, equitability and consistencyequitability and consistency

Ian JolliffeIan Jolliffe

Forecast verificationHedging

ProprietyEquitabilityConsistency

22

Forecast verificationForecast verification

• Forecasts of weather and climate are Forecasts of weather and climate are made at all timescales.made at all timescales.

• It is desirable to verify/validate/assess It is desirable to verify/validate/assess forecastsforecasts– Science: A look back at the accuracy of Science: A look back at the accuracy of

forecasts is necessary to determine whether forecasts is necessary to determine whether current forecasting methods should be current forecasting methods should be continued, abandoned or modified.continued, abandoned or modified.

– Users: Decisions are based on past data but Users: Decisions are based on past data but also on forecasts of data not yet observed.also on forecasts of data not yet observed.

33

Verification measuresVerification measures

• In a In a measures-oriented approach measures-oriented approach toto forecast forecast verification, the value of one or more measure verification, the value of one or more measure that quantifies desirable properties of a set of that quantifies desirable properties of a set of forecasts is calculated. forecasts is calculated.

• Often a variety of measures or scores is Often a variety of measures or scores is available for a set of forecasts. To choose available for a set of forecasts. To choose between them, desirable properties of the between them, desirable properties of the measures themselves have been defined. This measures themselves have been defined. This is called is called metaverification. metaverification.

• Such criteria include propriety, consistency Such criteria include propriety, consistency and equitability, all of which relate to the and equitability, all of which relate to the avoidance of hedging.avoidance of hedging.

44

HedgingHedging

• Hedging has a variety of definitions, but is Hedging has a variety of definitions, but is commonly taken in everyday use to mean ‘placing commonly taken in everyday use to mean ‘placing bets on the opposite side in order to cut losses or bets on the opposite side in order to cut losses or guarantee a minimum amount of winnings’. In guarantee a minimum amount of winnings’. In other words a forecast allows more than one other words a forecast allows more than one (conflicting) possibility.(conflicting) possibility.

• The term is fairly well-known in meteorology, The term is fairly well-known in meteorology, though not very often used in print. When it is, it is though not very often used in print. When it is, it is taken to mean that it occurs (Murphy, 1978) taken to mean that it occurs (Murphy, 1978) ‘whenever a forecaster’s judgement and forecast ‘whenever a forecaster’s judgement and forecast differ’.differ’.

55

To hedge or not to hedgeTo hedge or not to hedge

• ‘‘A meteorologist who prepares probability A meteorologist who prepares probability forecasts should not “hedge,” i.e. the forecasts should not “hedge,” i.e. the meteorologist’s probabilities should express meteorologist’s probabilities should express his true beliefs’.his true beliefs’.

• ‘‘A meteorologist whose forecasts are A meteorologist whose forecasts are evaluated with a particular scoring system evaluated with a particular scoring system can, and should, be expected to “hedge” to can, and should, be expected to “hedge” to obtain the best possible score.’obtain the best possible score.’

• Both quotations express plausible positions. Both quotations express plausible positions. Both are from Murphy & Epstein (1967), the Both are from Murphy & Epstein (1967), the latter deriving from a panel discussion latter deriving from a panel discussion reported in BAMS (1952).reported in BAMS (1952).

66

Why hedge? Why hedge? • Hedging is used to make some sort of gain over Hedging is used to make some sort of gain over

what can be achieved without hedging.what can be achieved without hedging.• In everyday usage the gain is financial.In everyday usage the gain is financial.• In meteorology the gain is a better value or In meteorology the gain is a better value or

expected value of some score used to assess/verify expected value of some score used to assess/verify forecasts. Hence hedging is ‘playing the score’.forecasts. Hence hedging is ‘playing the score’.

• To make both quotations compatible we can To make both quotations compatible we can restrict ourselves to using scores for which hedging restrict ourselves to using scores for which hedging is impossible – we need proper, or consistent, or is impossible – we need proper, or consistent, or perhaps equitable, scores.perhaps equitable, scores.

• Caveat: If missing an extreme event is far more Caveat: If missing an extreme event is far more costly than a ‘false alarm’, hedging by over-costly than a ‘false alarm’, hedging by over-forecasting is actually desirable. This is because forecasting is actually desirable. This is because the concept of the concept of value value comes into play.comes into play.

77

Finley’s tornadosFinley’s tornados

Tornado Tornado ObserveObservedd

Tornado Tornado not not observedobserved

TotalTotal

Tornado Tornado Forecast Forecast

28 (a)28 (a) 72 (b)72 (b) 100100

Tornado Tornado not not forecast forecast

23 (c)23 (c) 2680 (d)2680 (d) 27032703

TotalTotal 5151 27522752 28032803

Finley’s classic tornado forecasts: Consider the measure Proportion Correct (PC) = (a+d)/n = 2708/2803 = 0.966.The naïve strategy of always forecasting ‘no tornado’ does better (2752/2803 = 0.981), so PC can be hedged.

88

Value of Tornado ForecastsValue of Tornado Forecasts

• If a wrong forecast of any sort costs $1K, then the If a wrong forecast of any sort costs $1K, then the cost of the forecasting system is $95K, but the naive cost of the forecasting system is $95K, but the naive system costs only $51K – so the forecast system system costs only $51K – so the forecast system should not be used.should not be used.

• However, if a false alarm costs $1K, but a tornado However, if a false alarm costs $1K, but a tornado missed costs $10K, then the system costs $302K, but missed costs $10K, then the system costs $302K, but naivety costs $510K. So the forecasting system saves naivety costs $510K. So the forecasting system saves $208K compared to naivety.$208K compared to naivety.

• … … not to mention lives. not to mention lives. • Following Finley’s article, Peirce (1884) not only Following Finley’s article, Peirce (1884) not only

devised an equitable score, but looked at ‘utility’, devised an equitable score, but looked at ‘utility’, associating a ‘profit’ with hits and ‘loss’ with false associating a ‘profit’ with hits and ‘loss’ with false alarms.alarms.

99

Propriety (proper scores)Propriety (proper scores)

• Next they should be taught about Next they should be taught about equitability and consistency.equitability and consistency.

1010

Propriety IIPropriety II• For probability forecasts, a (strictly) proper For probability forecasts, a (strictly) proper

scoring system is one for which the forecaster scoring system is one for which the forecaster obtains the best possible expected score by obtains the best possible expected score by forecasting his/her true beliefs (and only by forecasting his/her true beliefs (and only by doing so) – Murphy & Epstein, 1967.doing so) – Murphy & Epstein, 1967.

• It is not necessary to invoke a ‘true belief’ to It is not necessary to invoke a ‘true belief’ to argue for propriety – Broeker and Smith, 2007.argue for propriety – Broeker and Smith, 2007.

• The Brier score is the best known proper score The Brier score is the best known proper score – there are others (logarithmic, spherical), – there are others (logarithmic, spherical), also plenty of theory and discussion. Equally, also plenty of theory and discussion. Equally, many scores (e.g. linear) are not proper.many scores (e.g. linear) are not proper.

1111

Propriety is irrelevant for Propriety is irrelevant for deterministic forecastsdeterministic forecasts

EitherEither– The forecaster’s true belief is The forecaster’s true belief is

probabilistic.probabilistic.

OrOr– The forecast is certain his/her The forecast is certain his/her

(deterministic) forecast is correct, so the (deterministic) forecast is correct, so the best possible value of the score will be best possible value of the score will be achieved for any score, and there is no achieved for any score, and there is no incentive to hedge.incentive to hedge.

1212

EquitabilityEquitability

• ‘‘All’ unskilled forecasts should have the same All’ unskilled forecasts should have the same expected score.expected score.

• Not so obviously related to hedging as Not so obviously related to hedging as propriety. propriety.

• But if a score is not equitable, it can be hedged But if a score is not equitable, it can be hedged in the sense that a forecaster who knows (s)he in the sense that a forecaster who knows (s)he has little skill may do better using an unskilled has little skill may do better using an unskilled forecast with a better expected score. forecast with a better expected score.

1313

Equitability – what is an Equitability – what is an ‘unskilled’ forecast?‘unskilled’ forecast?

1.1. Choosing a forecast by some Choosing a forecast by some completely random mechanism.completely random mechanism.

2.2. Always making the same forecast (but Always making the same forecast (but what if that forecast is ‘climatology’?)what if that forecast is ‘climatology’?)

3.3. Forecasting persistence??Forecasting persistence??

Most definitions include 1 & 2, but not 3. Most definitions include 1 & 2, but not 3.

1414

Propriety and equitability Propriety and equitability

No scoring system for probability No scoring system for probability forecasts can be both proper and forecasts can be both proper and equitable. Given the choice, equitable. Given the choice, which would you prefer?which would you prefer?– Propriety or equitability?Propriety or equitability?

1515

Equitability and probability Equitability and probability forecasts forecasts • Not only is equitability incompatible with Not only is equitability incompatible with

propriety, but it is rather difficult to achieve propriety, but it is rather difficult to achieve equitability at all for probability forecasts. equitability at all for probability forecasts.

• Any score is a function S(d) of the difference d=f-Any score is a function S(d) of the difference d=f-o, where f is forecast probability and o the o, where f is forecast probability and o the corresponding observation, which is always 0 or 1.corresponding observation, which is always 0 or 1.

• If S(d) is required to be symmetric [S(d)=S(-d)], If S(d) is required to be symmetric [S(d)=S(-d)], then equitability is impossible unlessthen equitability is impossible unless– S(p) = S(1-p) for all p between 0 and 1 S(p) = S(1-p) for all p between 0 and 1 oror– The base rate/climatology The base rate/climatology θθ is equal to 0.5. is equal to 0.5.

• The second of these conditions is very restrictive The second of these conditions is very restrictive and the first makes no sense.and the first makes no sense.

1616

Equitability and probability Equitability and probability forecasts IIforecasts II

• What about allowing non-symmetry of S(d)? It is then What about allowing non-symmetry of S(d)? It is then relatively easy to get an equitable score.relatively easy to get an equitable score.

• Arbitrarily set S(0) = 0, S(1) =1 (other ‘anchor points’ Arbitrarily set S(0) = 0, S(1) =1 (other ‘anchor points’ give qualitatively similar conclusions).give qualitatively similar conclusions).

• The nature of the asymmetry is determined by the base The nature of the asymmetry is determined by the base rate rate θ.θ.– For For θ = 1/3, S(1) = 1 and S(-1) = 2.θ = 1/3, S(1) = 1 and S(-1) = 2.– For θ = 0.1, S(1) = 1 and S(-1) = 9.For θ = 0.1, S(1) = 1 and S(-1) = 9.

• It is plausible that a highly asymmetric score might be It is plausible that a highly asymmetric score might be needed when ‘value’ is taken into account; much less so needed when ‘value’ is taken into account; much less so that the asymmetry is determined solely by base rate …that the asymmetry is determined solely by base rate …

• But perhaps penalising wrong forecasts a long way from But perhaps penalising wrong forecasts a long way from base rate less than wrong forecasts close to base rate base rate less than wrong forecasts close to base rate has some merit??has some merit??

1717

ConsistencyConsistency

• For deterministic forecasts ‘consistency’ takes For deterministic forecasts ‘consistency’ takes the place of ‘propriety’.the place of ‘propriety’.

• Like ‘hedging’ the meaning is slightly different Like ‘hedging’ the meaning is slightly different from everyday usage.from everyday usage.

• There are ‘consistent’ forecasts – those that There are ‘consistent’ forecasts – those that correspond with the forecaster’s judgments i.e. correspond with the forecaster’s judgments i.e. the forecaster does not hedge (Murphy, 1993).the forecaster does not hedge (Murphy, 1993).

• There are also ‘consistent’ performance There are also ‘consistent’ performance measures (scores) …measures (scores) …

1818

Consistent scoresConsistent scores• For the definition of consistency given by For the definition of consistency given by

Murphy & Daan (1985) we need to assume Murphy & Daan (1985) we need to assume that any forecaster really has a probability that any forecaster really has a probability distribution for the variable to be forecast and distribution for the variable to be forecast and that a rule or directive determines the that a rule or directive determines the deterministic forecast to be made, given the deterministic forecast to be made, given the forecaster’s probability distribution.forecaster’s probability distribution.

• Then a score is consistent with the directive if Then a score is consistent with the directive if that score is minimised by forecasting using that score is minimised by forecasting using that directive. For example if the directive is that directive. For example if the directive is ‘forecast the mean’ for a continuous variable, ‘forecast the mean’ for a continuous variable, then mean square error is a consistent score then mean square error is a consistent score as it is minimised by forecasting the mean. as it is minimised by forecasting the mean.

1919

Consistent scores - Consistent scores - remarksremarks

• The assumption of how the forecaster behaves (forecast The assumption of how the forecaster behaves (forecast deterministically when his/her beliefs are probabilistic) deterministically when his/her beliefs are probabilistic) implies that the forecaster always hedges in the sense that implies that the forecaster always hedges in the sense that true beliefs are not forecast.true beliefs are not forecast.

• However, it is the opposite of hedging in its everyday However, it is the opposite of hedging in its everyday usage (make forecasts less definite), and is not done to usage (make forecasts less definite), and is not done to improve a score.improve a score.

• As a slight digression, Murphy (1978) argues that ‘the As a slight digression, Murphy (1978) argues that ‘the desire to eliminate hedging should encourage forecasters desire to eliminate hedging should encourage forecasters to express … forecasts in probabilistic terms’. to express … forecasts in probabilistic terms’.

• It seems to me that the definition of consistency could be It seems to me that the definition of consistency could be turned around to say that a directive is consistent with a turned around to say that a directive is consistent with a score, rather than a score consistent with a directive. score, rather than a score consistent with a directive. Given any score, if we can find its consistent directive we Given any score, if we can find its consistent directive we can reverse the definition again to say that the score is can reverse the definition again to say that the score is consistent. consistent.

2020

Hedging for deterministic Hedging for deterministic forecastsforecasts

• First, what do we mean by hedging?First, what do we mean by hedging?– For probability forecasts it implies improving For probability forecasts it implies improving

‘expected score’. But expectation is with ‘expected score’. But expectation is with respect to the forecaster’s true beliefs.respect to the forecaster’s true beliefs.

• For deterministic forecasts either the For deterministic forecasts either the forecaster’s true beliefs are forecaster’s true beliefs are – Deterministic (and clearly wrong) Deterministic (and clearly wrong) oror– Probabilistic and unknown.Probabilistic and unknown.

• So does hedging now imply improving So does hedging now imply improving actual score? Or is there another actual score? Or is there another definition?definition?

2121

Back to equitability Back to equitability

• Although apparently not very useful for Although apparently not very useful for probability forecasts, equitability is often probability forecasts, equitability is often made a requirement for deterministic made a requirement for deterministic categorical forecasts.categorical forecasts.

• Does equitability rule out hedging when Does equitability rule out hedging when hedging implies improvement of actual score?hedging implies improvement of actual score?

• Does non-equitability necessarily imply that a Does non-equitability necessarily imply that a score can be hedged?score can be hedged?

2222

Equitability – a conjectureEquitability – a conjecture

• Equitability ensures that hedging is Equitability ensures that hedging is impossible for deterministic categorical impossible for deterministic categorical forecasts.forecasts.

• It works for the Peirce skill score, an It works for the Peirce skill score, an equitable score for binary forecastsequitable score for binary forecasts– Transferring a proportion of forecasts of an Transferring a proportion of forecasts of an

event to forecasts of no event or, conversely, event to forecasts of no event or, conversely, transferring a proportion of no-event forecasts transferring a proportion of no-event forecasts to ‘event’, reduces the Peirce skill score.to ‘event’, reduces the Peirce skill score.

– But is this the only way that a forecaster can But is this the only way that a forecaster can diverge from his/her true beliefs?diverge from his/her true beliefs?

2323

Non-equitable scoresNon-equitable scores

• Non-equitable scores may or Non-equitable scores may or may not be hedged, depending may not be hedged, depending on details of the dataon details of the data

• Consider ‘Proportion Correct’ in Consider ‘Proportion Correct’ in a (2x2) table, a non-equitable a (2x2) table, a non-equitable score, (a+d)/n score, (a+d)/n

• If a>b and d>c the score cannot If a>b and d>c the score cannot be improved by transferring a be improved by transferring a proportion of forecasts to non-proportion of forecasts to non-forecasts, or vice versa; forecasts, or vice versa; otherwise it can. For the otherwise it can. For the tornados, a < b, so PC can be tornados, a < b, so PC can be hedged.hedged.

ObservObserve e eventevent

Observe Observe no no eventevent

ForecasForecast eventt event aa bb

ForecasForecast no t no eventevent

cc dd

2424

Questions? Or answers!