Ed Tollerud, Tara Jensen, Barb Brown(also Yuejian Zhu, Zoltan Toth, Tony Eckel, Curtis Alexander, Huiling Yuan,…)

Module 6 Objective: Provide a portable, inclusive toolset for

Testing and Evaluation applications

Verification Module for the DTC Ensemble Testbed

DET Ensemble Workshop 2010

Wally Clark

http://esrl.noaa.gov/gsd/fab

Module 1: Configuration

Module 2: Initial

Perturbations

Module 4: Statistical Post-

Processing

External Input (HMT, HWT,

HFIP, etc)

Module 3: Module

Perturbations

Module 5: Product

Generation

Module 6: VerificationD

ET M

OD

ULES

Motivation and Requirements

Provide ensemble and probabilistic verification utilities that are of value to:

1. Model development2. Research applications 3. Operational feedback for forecasters4. DET tasks and user-driven projects

Develop readily applicable routines to:

1. Assemble and process ensemble forecasts2. Apply standard probabilistic verification algorithms3. Employ newly-developed evaluation methods4. Display results in thorough and intuitive fashion

DET Ensemble Workshop 2010

Questions for breakout groupEnsuring applicability/acceptability to user groups?

Motivation and Requirements (continued)

Module should include verification utilities that are:

Objective but user-driven (‘honest broker’) Comprehensive but portable Flexible and extensible User-friendly and supported (internet-enable) State-of-the-art (Object-based; spatial ensemble verification)

Utilities should be designed for:

Retrospective diagnostic studies and intensive forecast evaluation

Real-time application during quasi-operational field exercises (HWT and HMT experience)

DET Ensemble Workshop 2010

Questions for breakout groupHow to insure portability to different ensemble forecast systems and applications?Critical operational or real-time capabilities?Other usage types?

Verification Algorithms, Options, and Related Displays

DET Ensemble Workshop 2010

1) Comprehensive set of standard and probabilistic scores2) Why not all that are feasible?3) Examples of standard scores: MAE, ME, RMSE, ETS, POD,FAR, Bias,

Odds ratio, etc.4) Probabilistic scores and display types: Brier score and Brier skill scores

and decomposition (reliability and resolution), Rank and Continuous Rank Probability Scores, ROC curve, Talagrand Diagram, Rank Histogram

Questions for breakout groupHow to involve user group in contributions to displays and scoring algorithms? DET Visitor Program?

Examples: Individual member standard scores from HMT/HWT

DET Ensemble Workshop 2010

Examples: Brier Skill Score and Decomposition Products; Talagrand

Diagram (Zhu et al.); Dispersion Diagram (Tony Eckel)

Examples: New Display Products (RAL and NAEFS)

DET Ensemble Workshop 2010

Questions for breakout groupHow to develop and solicit new display possibilities? (Ensemble Products Workshop II?)

Verification Dataset Choices

DET Ensemble Workshop 2010

QPF verification example: QPE possibilities include:

Operational Gage datastreams Other Mesonet gage sets (CoCoRahs; OK mesonet; ….) Radar products (Stage IV; Q2;…) Satellite estimates from many sources

Start-up default for verification: NCEP/EMC verification datasets; e.g., RTMA, PrepBufr,….

Analysis of Record (AOR) as source of high-resolution data

Questions for breakout groupOffer data options? Develop standardized datasets? Supply QC applications? Incorporate MADIS? How to build in data uncertainty?

Verification Packages and Products to Incorporate

MET tools: Advantages include:1. Strong synergy with other DTC tasks (HMT; HWT; HFIP)2. In-house DTC product with good support3. Continual updates, revisions, and novel additions4. Evolving ensemble and probabilistic capabilities5. Portable display systems with wide use (r; ncl)6. ‘Metviewer’ interactive web display7. Replication of EMC capabilities8. Emphasis on confidence intervals and spatial methods

Other verification products and packages with valuable input:9. EMC probabilistic products10.OHD hydrologic verification products11. RUC and HRRR ensemble display products12.NEVS procedures for data access13. AWIPS; ALPS; NPVU; AFWA; others….

DET Ensemble Workshop 2010

Tentative Timeline and Milestones

DET Ensemble Workshop 2010

June 2010 – Initial probabilistic scoring utilities demonstrated aspart of HWT spring exercise

October 2010 – Verification module plan completed and reviewed by WRF ensemble working group

January 2011 – Expanded probabilistic scoring utilities demonstrated as part of HMT winter exercise

June 2011 – Prototype real-time web-based verification site with basic capabilities completedSeptember 2011 – Verification utility installed as full module in end-to-end workflow

Questions for breakout groupContribute to other workshops (i.e., RAL Verification Workshop 11/2010; WMO Verification tutorials; others?)

Useful Websites and Information Sources

DET Ensemble Workshop 2010

NPVU: http://www.hpc.ncep.noaa.gov/npvu/ESRL/AMB: http://ruc.noaa.gov/hcpf/hcpf_verif.cgi OHD: http://hydis8.eng.uci.edu/hepex/Workshops/postprocesswksp/ POSTERS/Demargne-33.pdfOHD: http://hydis8.eng.uci.edu/hepex/testbeds/Verification.htmEMC/NCEP: https://ams.confex.com/ams/pdfpapers/131645.pdf‘r’ Stats Pkg: http://cran.r-project.org/web/packages/verification/verification.pdfMET Verification example: http://verif.rap.ucar.edu/eval/hmt/2010/Others:

Questions for breakout groupHow to promote portability between user groups?

Breakout Group Discussion – User Community Issues

DET Ensemble Workshop 2010

1) Prioritized list of ensemble-relevant verification products

2) Dataset choices and quality

How to promote portability between user groups?

Questions for breakout groupEnsuring applicability to user groups?

How to insure portability to different ensemble forecast systems and applications?Critical operational or real-time capabilities?

How to involve user group in contributions to displays and scoring algorithms? DET Visitor Program?

How to develop and solicit new display possibilities? (Ensemble Products Workshop II?)

Offer data options? Develop standardized datasets? Supply QC applications? Incorporate MADIS? How to build in data uncertainty?

Contribute to other workshops (i.e., RAL Verification Workshop 11/2010; WMO Verification tutorials; others?)