© 2009 ucar. all rights reserved. atec-4dwx ipr, 21−22 april 2009 national security applications...

© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009

National Security Applications Program Research Applications Laboratory

Ensemble-4DWX update: focus on calibration and verification



Summary of progress

• Upgraded from WRF V2.2.1 to V3.0.1.1• Replaced 3 physics members (slab LSM, Grell-Dvevenyi Cu, and

Betts-Miller Cu schemes); added RUC LSM, no horizontal diffusion, Thomson scheme with positive-definite advection

• Configured E-4DWX for ATC and operated for a 2 month period (Dec 1 and Jan 30)

• Configured and run E-4DWX for supporting UT Dept of Air Quality• Improved MYJ and YSU PBL height diagnosis and PBL mixing• Added new graphics and improve post-processing flexibility

(installation for GMOD and plotting historical case archive) and computing parallelisms

• Presented the system at a number of AMS and other conferences• Continued R&D of on-line verification and calibration



Ongoing work and plans

• E-4DWX science paper for MWR; technology brief for BAMS

• Member performance evaluation for 2009 Feb and Mar, and compare to the results we obtained for 2008 Feb and Mar; look for improvements

• Member-based evaluation of ATC E-4DWX run to identify best WRF model configuration

• ATC cold-air damming study using E-4DWX archive

• Begin development of 4d-ENKF



New product examples

© 2009 UCAR. All rights reserved.


ATEC-4DWX IPR, 21−22 April 2009 5


© 2009 UCAR. All rights reserved.


ATEC-4DWX IPR, 21−22 April 2009 6




Calibration and verification

• Ensemble calibration to correct predicted distribution.

• Calibration is needed for users capable of decision making with probabilistic guidance. Will be needed for foreseeable future.

• Verification of different ensemble characteristics is easily completed when performing calibration.

NCAR/RAL - National Security Applications Program8

What do we mean by “calibration” or “post-processing”?

Pro

babi

lity

calibration

Temperature [K]

Pro

babi

lity

Temperature [K]

Post-processing has corrected:• the “on average” bias• as well as under-representation of the 2nd moment of the empirical forecast PDF (i.e. corrected its “dispersion” or “spread”)

“spread” or “dispersion”

“bias”obs

obs

ForecastPDF

ForecastPDF


Benefits of Post-Processing

Essential for tailoring to local application:

NWP provides spatially- and temporally-averaged gridded forecast output

=> Applying gridded forecasts to point locations requires location specific calibration to account for spatial- and temporal- variability ( => increasing ensemble dispersion)

=> Relatively inexpensive!


Example of Quantile Regression (QR)

Our application

Fitting T quantiles using QR conditioned on:

1) Reforecast ens

2) ensemble mean

3) ensemble median

4) ensemble stdev

5) Persistence

6) Log Reg quantile



Summary of progress

• Calibration (and Verification) Scheme fully-automated• Calibrating Temperature and Dew Point Temperature• Calibration specific for unique set of models that are available for

each cycle• Utilizes “persistence” if available• 26 sites over DPG• Full calibration for all sites ~ 10hrs for each weather variable• Using lookup tables ~ 1hr• Updating tables once per week


Calibration Procedure1) Fit Logistic Regression (LR) ensembles

– Calibrate CDF over prescribed set of climatological quantiles

– For each forecast: resample 15 member ensemble set

For each quantile:

2) Perform a “climatological” fit to the data

3) Starting with full regressor set, iteratively select best subset using “step-wise cross-validation”

– Fitting done using QR– Selection done by:

a) Minimizing QR cost functionb) Satisfying the binomial distribution

( 2nd pass: segregate forecasts into differing ranges of ensemble dispersion, and refit models )

Pro

babi

lity

Temperature [K]

obs ForecastPDF

T [

K]

TimeForecastsobserved

Regressors for each quantile: 1) reforecast ensemble 2) ens mean 3) ens median 4) ens stdev 5) persistence 6) logistic regression quantile


Verifying ensemble forecasts

Measures Used in automated model selection:

1)Rank histogram2)Root Mean square error (RMSE)3)Brier score4)Rank Probability Score (RPS)5)Relative Operating Characteristic (ROC) curve



6hr Temperature Time-series

Before Calibration After Calibration




Before Calibration After Calibration


Raw versus Calibrated PDF’s

obs

Blue is “raw” ensembleBlack is calibrated ensembleRed is the observed value

Notice: significant change in both “bias” and dispersion of final PDF

(also notice PDF asymmetries)

Troubled Rank Histograms

Slide from Matt Pocernic

1 2 3 4 5 6 7 8 9 10Ensemble #

1 2 3 4 5 6 7 8 9 10Ensemble #

Co

un

ts0

1020

30

Co

un

ts0

1020

30



6hr Temperature Rank Histograms



36hr Temperature Rank Histograms


Verifying ensemble forecasts

Measures Used:

1)Rank histogram2)Root Mean square error (RMSE)3)Brier score4)Rank Probability Score (RPS)5)Relative Operating Characteristic (ROC) curve

=> Using these for automated calibration model selection


RPS =1

n−1CDFfc,i −CDFobs,i( )

2

i=1

n

∑

Rank Probability Scorefor multi-categorical or continuous variables

Skill Scores

• Single value to summarize performance.

• Reference forecast - best naive guess; persistence, climatology

• A perfect forecast implies that the object can be perfectly observed

• Positively oriented – Positive is good

SS =Aforc −ArefAperf −Aref




RMSE Skill Score CRPS Skill Score

Reference Forecasts:Black -- raw ensembleBlue -- persistence



RMSE of Models

6hr Lead-time 36hr Lead-time



Significant Calibration Regressors

6hr Lead-time 36hr Lead-time


Future Plans

Correct over-dispersion of calibration

Implement procedure for a) wind speed, b) wind direction, c) precipitation, d) pressure

Diagnose most informative model set to use operationally

Develop Scheme for model points without surface observations

=> over whole model-gridded domain

Continuous scores: MSE

( )∑=

−=n

iii xy

nMSE

1

21

Average of the squares of the errors: it measures the magnitude of the error, weighted on the squares of the errors

it does not indicate the direction of the error

Quadratic rule, therefore large weight on large errors: good if you wish to penalize large error sensitive to large values (e.g. precipitation) and outliers; sensitive to large variance (high resolution models); encourage conservative forecasts (e.g. climatology)

Attribute:

measures

accuracy

Slide from Barbara Casati

=> For ensemble forecast, use ensemble mean

Scatter-plot and

Contingency Table

Does the forecast detect correctly temperatures above 18 degrees ?


BS =1n

yi −oi( )2

i=1

n

∑

Brier Score

y = forecasted event occurenceo = observed occurrence (0 or 1)i = sample # of total n samples

=> Note similarity to MSE

Conditional Distributions

Conditional histogram and conditional box-plot


Scatter-plot and Contingency TableDoes the forecast detect correctly temperatures above 18 degrees ?

Does the forecast detect correctly temperatures below 10 degrees ?


Discrimination Plot

Outcome = No

False Alarms

Outcome = Yes

Hits

Decision Threshold



Receiver Operating Characteristic (ROC) Curve

© 2009 ucar. all rights reserved. atec-4dwx ipr, 21−22 april 2009 national security applications...

Documents