most popular “traditional statistics”: ets, bias problem: what does the ets tell us ?
DESCRIPTION
Bias Adjusted Precipitation Scores Fedor Mesinger NOAA/Environmental Modeling Center and Earth System Science Interdisciplinary Center (ESSIC), Univ. Maryland, College Park, MD VX-Intercompare Meeting Boulder, 20 February 2007. Most popular “traditional statistics”: ETS, Bias - PowerPoint PPT PresentationTRANSCRIPT
Bias Adjusted Precipitation Scores
Fedor Mesinger
NOAA/Environmental Modeling Centerand
Earth System Science Interdisciplinary Center(ESSIC), Univ. Maryland, College Park, MD
VX-Intercompare MeetingBoulder, 20 February 2007
Most popular “traditional statistics”:
ETS, Bias
Problem: what does the ETS tell us ?
“The higher the value, the better the model skill is for the particular threshold”
(a recent MWR paper)
Example:Three models, ETS, Bias, 12 months, “Western Nest”
Is the green model loosing to red because of a bias penalty?
What can one do ?
BIAS NORMALIZED PRECIPITATION SCORES
Fedor Mesinger1 and Keith Brill2
1NCEP/EMC and UCAR, Camp Springs, MD2NCEP/HPC, Camp Springs, MD
J12.617th Prob. Stat. Atmos. Sci.; 20th WAF/16th NWP (Seattle AMS, Jan. ‘04)
Two methods of the adjustment for bias(“Normalized” not the best idea)
1. dHdF method: Assume incremental change in hits per incremental change in bias is proportional to the “unhit” area, O-
H
Objective: obtain ETS adjusted to unit bias, to show the model’s accuracy in placing
precipitation(The idea of the adjustment to unit bias to arrive at placement accuracy:
Shuman 1980, NOAA/NWS Office Note)
2. Odds Ratio method: different objective
O
H
a
b
c
d
F
Forecast, Hits, and Observed (F, H, O) area, or number of model grid boxes:
dHdF method, assumption:
can be solved;
a function H (F) obtained that satisfies the three
requirements:€
dHdF
= a (O − H ), a = const,
• Number of hits H -> 0 for F -> 0;
• The function H(F) satisfies the
known value of H for the model’s F,
the pair denoted by Fb, Hb, and,
• H(F) -> O as F increases
West
EtaGFS
NMM
Bias adjusted eq. threats
A downside: if Hb is close to Fb, or to O,it can happen that
dH/dF > 1 for F -> 0
Physically unrealistic !
Reasonableness requirement:
€
Hb ≤ O(1−e−Fb /O )
“dHdM”method:
O
H
a
b
c
d
F
Assume as F is increased by dF, ratio of the infinitesimal increase in H, dH, and that in
false alarms dM=dF-dH, is proportional to the yet unhit area:
€
dH
dM= b(O − H)
€
b = const
One obtains
( Lambertw, or ProductLog in Mathematica,is the inverse function of
€
z = wew )
€
H(F) = O −1
blambertw bOeb(O−F )
( )
H (F) now satisfies the additional requirement:
dH/dF never > 1
20 40 60 80 100 120 140
F
20
40
60
80
100
120H
H(F)
H = O
H = F
Fb , Hb
dHdF method
H(F)
H = O
H = F
Fb , Hb
20 40 60 80 100 120 140F
20
40
60
80
100
120H
dHdM method
Results for the two “focus cases”,dHdM method
(Acknowledgements: John Halley Gotway, data; Dušan Jović, code and
plots)
5/13 Case dHdM
wrf2capswrf4ncarwrf4ncep
6/01 Case
dHdM
wrf2capswrf4ncarwrf4ncep
Impact, in relative terms, for the two cases is small, because the biases of the three models
are so similar !
One more case, for good measure:
5/25 Case dHdM
wrf2capswrf4ncarwrf4ncep
Comment:
Scores would have generally been higher had the verification been done on grid squares
greater than ~4 km
This would have amounted to a poor-person’s version of “fuzzy” methods !