© 2009 ucar. all rights reserved. atec-4dwx ipr, 21−22 april 2009 national security applications...
Post on 18-Jan-2016
214 Views
Preview:
TRANSCRIPT
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
Ensemble-4DWX update: focus on calibration and verification
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
Summary of progress
• Upgraded from WRF V2.2.1 to V3.0.1.1• Replaced 3 physics members (slab LSM, Grell-Dvevenyi Cu, and
Betts-Miller Cu schemes); added RUC LSM, no horizontal diffusion, Thomson scheme with positive-definite advection
• Configured E-4DWX for ATC and operated for a 2 month period (Dec 1 and Jan 30)
• Configured and run E-4DWX for supporting UT Dept of Air Quality• Improved MYJ and YSU PBL height diagnosis and PBL mixing• Added new graphics and improve post-processing flexibility
(installation for GMOD and plotting historical case archive) and computing parallelisms
• Presented the system at a number of AMS and other conferences• Continued R&D of on-line verification and calibration
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
Ongoing work and plans
• E-4DWX science paper for MWR; technology brief for BAMS
• Member performance evaluation for 2009 Feb and Mar, and compare to the results we obtained for 2008 Feb and Mar; look for improvements
• Member-based evaluation of ATC E-4DWX run to identify best WRF model configuration
• ATC cold-air damming study using E-4DWX archive
• Begin development of 4d-ENKF
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
New product examples
© 2009 UCAR. All rights reserved.
New product examples
ATEC-4DWX IPR, 21−22 April 2009 5
National Security Applications Program Research Applications Laboratory
© 2009 UCAR. All rights reserved.
New product examples
ATEC-4DWX IPR, 21−22 April 2009 6
National Security Applications Program Research Applications Laboratory
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
Calibration and verification
• Ensemble calibration to correct predicted distribution.
• Calibration is needed for users capable of decision making with probabilistic guidance. Will be needed for foreseeable future.
• Verification of different ensemble characteristics is easily completed when performing calibration.
NCAR/RAL - National Security Applications Program8
What do we mean by “calibration” or “post-processing”?
Pro
babi
lity
calibration
Temperature [K]
Pro
babi
lity
Temperature [K]
Post-processing has corrected:• the “on average” bias• as well as under-representation of the 2nd moment of the empirical forecast PDF (i.e. corrected its “dispersion” or “spread”)
“spread” or “dispersion”
“bias”obs
obs
ForecastPDF
ForecastPDF
NCAR/RAL - National Security Applications Program9
Benefits of Post-Processing
Essential for tailoring to local application:
NWP provides spatially- and temporally-averaged gridded forecast output
=> Applying gridded forecasts to point locations requires location specific calibration to account for spatial- and temporal- variability ( => increasing ensemble dispersion)
=> Relatively inexpensive!
NCAR/RAL - National Security Applications Program10
Example of Quantile Regression (QR)
Our application
Fitting T quantiles using QR conditioned on:
1) Reforecast ens
2) ensemble mean
3) ensemble median
4) ensemble stdev
5) Persistence
6) Log Reg quantile
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
Summary of progress
• Calibration (and Verification) Scheme fully-automated• Calibrating Temperature and Dew Point Temperature• Calibration specific for unique set of models that are available for
each cycle• Utilizes “persistence” if available• 26 sites over DPG• Full calibration for all sites ~ 10hrs for each weather variable• Using lookup tables ~ 1hr• Updating tables once per week
NCAR/RAL - National Security Applications Program12
Calibration Procedure1) Fit Logistic Regression (LR) ensembles
– Calibrate CDF over prescribed set of climatological quantiles
– For each forecast: resample 15 member ensemble set
For each quantile:
2) Perform a “climatological” fit to the data
3) Starting with full regressor set, iteratively select best subset using “step-wise cross-validation”
– Fitting done using QR– Selection done by:
a) Minimizing QR cost functionb) Satisfying the binomial distribution
( 2nd pass: segregate forecasts into differing ranges of ensemble dispersion, and refit models )
Pro
babi
lity
Temperature [K]
obs ForecastPDF
T [
K]
TimeForecastsobserved
Regressors for each quantile: 1) reforecast ensemble 2) ens mean 3) ens median 4) ens stdev 5) persistence 6) logistic regression quantile
NCAR/RAL - National Security Applications Program13
Verifying ensemble forecasts
Measures Used in automated model selection:
1)Rank histogram2)Root Mean square error (RMSE)3)Brier score4)Rank Probability Score (RPS)5)Relative Operating Characteristic (ROC) curve
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
6hr Temperature Time-series
Before Calibration After Calibration
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
36hr Temperature Time-series
Before Calibration After Calibration
NCAR/RAL - National Security Applications Program16
Raw versus Calibrated PDF’s
obs
Blue is “raw” ensembleBlack is calibrated ensembleRed is the observed value
Notice: significant change in both “bias” and dispersion of final PDF
(also notice PDF asymmetries)
Troubled Rank Histograms
Slide from Matt Pocernic
1 2 3 4 5 6 7 8 9 10Ensemble #
1 2 3 4 5 6 7 8 9 10Ensemble #
Co
un
ts0
1020
30
Co
un
ts0
1020
30
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
6hr Temperature Rank Histograms
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
36hr Temperature Rank Histograms
NCAR/RAL - National Security Applications Program20
Verifying ensemble forecasts
Measures Used:
1)Rank histogram2)Root Mean square error (RMSE)3)Brier score4)Rank Probability Score (RPS)5)Relative Operating Characteristic (ROC) curve
=> Using these for automated calibration model selection
NCAR/RAL - National Security Applications Program21
RPS =1
n−1CDFfc,i −CDFobs,i( )
2
i=1
n
∑
Rank Probability Scorefor multi-categorical or continuous variables
Skill Scores
• Single value to summarize performance.
• Reference forecast - best naive guess; persistence, climatology
• A perfect forecast implies that the object can be perfectly observed
• Positively oriented – Positive is good
SS =Aforc −ArefAperf −Aref
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
36hr Temperature Time-series
RMSE Skill Score CRPS Skill Score
Reference Forecasts:Black -- raw ensembleBlue -- persistence
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
RMSE of Models
6hr Lead-time 36hr Lead-time
© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, 21−22 April 2009
National Security Applications Program Research Applications Laboratory
Significant Calibration Regressors
6hr Lead-time 36hr Lead-time
NCAR/RAL - National Security Applications Program26
Future Plans
Correct over-dispersion of calibration
Implement procedure for a) wind speed, b) wind direction, c) precipitation, d) pressure
Diagnose most informative model set to use operationally
Develop Scheme for model points without surface observations
=> over whole model-gridded domain
Continuous scores: MSE
( )∑=
−=n
iii xy
nMSE
1
21
Average of the squares of the errors: it measures the magnitude of the error, weighted on the squares of the errors
it does not indicate the direction of the error
Quadratic rule, therefore large weight on large errors: good if you wish to penalize large error sensitive to large values (e.g. precipitation) and outliers; sensitive to large variance (high resolution models); encourage conservative forecasts (e.g. climatology)
Attribute:
measures
accuracy
Slide from Barbara Casati
=> For ensemble forecast, use ensemble mean
Scatter-plot and
Contingency Table
Does the forecast detect correctly temperatures above 18 degrees ?
Slide from Barbara Casati
BS =1n
yi −oi( )2
i=1
n
∑
Brier Score
y = forecasted event occurenceo = observed occurrence (0 or 1)i = sample # of total n samples
=> Note similarity to MSE
Conditional Distributions
Conditional histogram and conditional box-plot
Slide from Barbara Casati
Scatter-plot and Contingency TableDoes the forecast detect correctly temperatures above 18 degrees ?
Does the forecast detect correctly temperatures below 10 degrees ?
Slide from Barbara Casati
Discrimination Plot
Outcome = No
False Alarms
Outcome = Yes
Hits
Decision Threshold
Slide from Matt Pocernic
Slide from Matt Pocernic
Receiver Operating Characteristic (ROC) Curve
top related