Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and

Temperature Forecasts

Tom HopsonLuca Delle Monache, Yubao Liu, Gregory Roux,

Wanli Wu, Will Cheng, Jason Knievel, Sue Haupt

Army Test and Evaluation Command:Dugway Proving Ground

Dugway Proving Grounds, Utah e.g. T Thresholds

• Includes random and systematic differences between members.

• Not an actual chance of exceedance unless calibrated.

Xcel Energy Service Areas

Wind Farms (50+)~3200 MW

Northern States Power (NSP)Public Service of Colorado (PSCO)Southwestern Public Service (SPS)

3.4 million customers (electric)Annual revenue $11B

Copyright 2010 University Corporation for Atmospheric Research

WRF RTFDDA Model DomainsEnsemble System (30 members)

D1 = 30 kmD2 = 10 km

0-48 hrs0-48 hrs

Real Time Four Dimensional Data Assimilation (RTFDDA)

41 vertical levels

Vary:•Multi-models•Lateral B.Cs.•Model Physics•External forcing

Yubao Liu -- yliu@ucar.edu for further questions

Goals of an EPS

• Predict the observed distribution of events and atmospheric states

• Predict uncertainty in the day’s prediction• Predict the extreme events that are possible on a

particular day• Provide a range of possible scenarios for a

particular forecast

OutlineI. Brief overview of: 1) quantile regression (QR),

2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog Kalman filter” (ANKF)

II. 2nd moment calibration via rank histogramsIII. Skill score comparisons and improvements

with increased hindcast dataIII. Example of blending approachesIV. Conclusions

Example of Quantile Regression (QR)

Our application

Fitting T quantiles using QR conditioned on:

1) Ranked forecast ens

2) ensemble mean

3) ensemble median

4) ensemble stdev

5) Persistence

Hopson and Hacker 2012

Logistic Regression for probability of exceedance (climatological thresholds)

f (z) =1

1+ e−z

z=β0 + β1x1 + ...+ βkxk

Prob

abili

ty/°

K

Temperature [K]

Prob

abili

ty/°

K

Temperature [K]

ForecastPDF

climatologicalPDF

Step I: determineclimatological quantiles

Prob

abili

ty

Temperature [K]

Step 3: use conditioned CDF tointerpolate desired quantiles

1.0

0.5

.75

.25

prior

posterior

Step 2: calculate conditionalprobs for each climat quan

Prob

abili

ty

Temperature [K]

0.5

.75

.25

1.0

Final result: “sharper” posterior PDFrepresented by interpolated quans

Hopson and Hacker 2012

T [K

]

Timeforecastsobserved

Regressor set: 1. reforecast ens2. ens mean3. ens stdev 4. persistence 5. LR quantile (not shown)

Prob

abili

ty/°

K

Temperature [K]

climatologicalPDF

Step I: Determineclimatological quantiles

Step 2: For each quan, use forward step-wisecross-validation to select best regress setSelection requires: a) min QR cost function, b) binomial distrib at 95% confidenceIf requirements not met, retain climatological “prior”

1.

3.2.

4.

Step 3: segregate forecasts based on ens dispersion; refit models (Step 2) for each range

Time

forecasts

T [K

]

I. II. III. II. I.Pr

obab

ility

/°K

Temperature [K]

ForecastPDF

prior

posterior

Final result: “sharper” posterior PDFrepresented by interpolated quans

Hopson and Hacker 2012

National Security Applications Program Research Applications Laboratory

Significant calibration regressors

3hr Lead-time 42hr Lead-time

Station DPG S01

National Security Applications Program Research Applications Laboratory

RMSE of ensemble members

3hr Lead-time 42hr Lead-time

Station DPG S01

Timet = 0day-1day-2day-6 day-5 day-4 day-3day-7

OBS

PRED

KF-weight

KF

Delle Monache et al. 2010

Analog Kalman Filter (ANKF)• Deterministic method applied to each individual ensemble

• KF weighting run in analog space

Timet = 0day-1day-2day-6 day-5 day-4 day-3day-7

OBS

PRED

KF-weight

KF

ANKF

AN

“Analog” Spaceday-4day-7day-5 day-3 day-2 day-1day-6

PRED

OBS

farthest analog

closest analog

NOTEThis procedure is applied independently at each

observation location and for a given forecast time

Delle Monache et al. 2010

OutlineI. Brief overview of: 1) quantile regression (QR),

2) logistic regression (LR), 3) umbrella post-processing procedure, 4) “analog Kalman filter” (ANKF)

II. 2nd moment calibration via rank histogramsIII. Skill score comparisons and improvements

with increased hindcast dataIII. Example of blending approachesIV. Conclusions

42-hr dewpoint time seriesBefore Calibration After Calibration (QR)

Station DPG S01

Original ensemble QR

LR ANKF

Rank Histograms15hr lead wind forecasts

Skill measures used:1) Rank histogram (converted to scalar measure)2) Root Mean square error (RMSE)3) Rank Probability Score (RPS)4) Relative Operating Characteristic (ROC) curve

Skill Scores

SS =Aforc −ArefAperf −Aref

Comparing to original ensemble forecast, but with bias removed => “reference forecast”

Blue - QR

Red - ANKF

Green - LR

Skill Score ComparisonFor wind farm CEDC, 3hr lead forecasts

Reference forecast: original wind speed ensemble w/ bias removedData size: 900pts

Rank Histogram scalar

QR QR

LR LR

RMSE

Skill Scores Dependence on Training Data Size

Upper dashed – 900ptsSolid – 600ptsLower dashed – 300pts

Reference Forecast:Original wind speed ensemble w/ bias removed

ROC

QR QR

LR LR

RPSS

Skill Scores Dependence on Training Data Size (cont)

Reference Forecast:Original wind speed ensemble w/ bias removed

Upper dashed – 900ptsSolid – 600ptsLower dashed – 300pts

23

RPS ROC

RMSE Brier Score

Wind farm TWBT

1

1 1

112

12 12

1224

24 24

2436

36 36 48

4848

48

36ANKFQRQR + ANKF

24

original ANKF

QR QR + ANKF

TWBT6-h

Summary “step-wise cross-validation”-based post-processing framework provides a method to ensure forecast skill no worse than climatological and persistence

Also provides an umbrella to blend together multiple post-processing approaches as well as multiple regressors, and to diagnose their utility for a variety of cost functions

Quantile regression and logistic regression useful tools for improving 2nd moment of ensemble distributions

See significant skill gains with increasing “hindcast data” amount for a variety of skill measures

Blending of post-processing approaches can also further enhance final forecast skill (e.g. ANKF and QR) by capturing “best of both worlds”

Further questions: hopson@ucar.edu or yliu@ucar.edu