1

Agenda Topic: Verification Strategy (Atmosphere)

Presented By: Geoff DiMego (NCEP/EMC)Contributors:

Stan Benjamin, Stephen Weygandt, Bonny Strong (ESRL/GSD) Fanglin Yang (IMSG at NCEP/EMC)

Ying Lin (NCEP/EMC), Tara Jensen (NCAR/DTC) Perry Shafran, Binbin Zhou (IMSG at NCEP/EMC)

Guang Ping Lou, Tracey Dorian (IMSG at NCEP/EMC) Keith Brill (IMSG at NCEP/WPC)

2

Operational System Attributes-1System Name Acronym Areal

CoverageHorz Res

Cycle Freq Fcst Length

Grid versus observation: Verifies NAM, NAM nests, FireWx nest, HiResWindow, DGEX, RAP, HRRR, SREF, CMAQ ozone & PM2.5, and GFS plus developmental parallels of all models

grid2obs Various 3-km thru 12-km

Hourly for NAM/nests/HRRR, RAP, 3-hourly for all others

15 hr thru 87 hr

Grid versus grid (gridded truth) including single and ensemble

grid2gid Global, CONUS, Alaska

Various Various Various

QPF verification pcpverif ConUS: 24h/6h FSS; 24h/3h SL1L2 and contingency table-based (FHO) scores. AK, HI, PR: 24h SL1L2 and FHO.

Multiple horizontalgrids ranging from 4.8km to 80km

24h, 6h, 3h 84h for NAM, up to 192h for GFS and DGEX. Other models: to the extent of forecast length or available data (in case of int’l models)

Cyclone track: Using model output data including analyses and forecasts. The models include, GFS, GEFS, NAM, SREF, CMC, CENS, ECMWF, EENS, NVGM, FENS, UKMET

Tracker Global & regional

Various

2- & 4-per day

84hr-180hr

3

Operational System Attributes-2

System Name Acronym Areal Coverage Horz Res

Cycle Freq

Fcst Length

Grid versus observation (for global NWP models, including GFS and ECMWF)

grd2ob NH, SH, Tropics, and CONUS

1.0-deg 4/day 168hr

Grid versus grid (gridded truth)(for global NWP models, see http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb)

grid2gid NH, SH, Tropics, and CONUS

2.5-deg

4/day 384hr

QPF verification(for global NWP models, including objected-oriented and ETS scores)

MODE CONUS 13km 4/day 168hr

Cyclone track (for global NWP models) hurtrack Atlantic, Western Pacific, Eastern Pacific

0.25-deg 4/day 126hr

GFS/GDAS data assimilation monitoring DA global various 4/day 6hr

4

Why System(s) are Operational

Primary stakeholders and requirement drivers• Developers and users both internal and external• Operational system decision makers, funding / portfolio / system managers• Research community (potential code contributors), private sector, universities

What products are the verification tools contributing to?• Improvement of NWP systems via long-term & comparative record of performance

• Myriad of stats, verification statistic database (VSDB), MySQL database• Web pages & display using FVS, Perl-based web, GRADS, and/or METviewer

What product aspects are you trying to improve with your development plans? • Enhance, streamline & simplify 3 components of verification: 1. calculation of error/skill

scores; 2. database storage & retrieval; 3. Web display (static images and interactive capability)• Community code model [METviewer at EMC and flexible html viewer (MATS = Model

Assessment Tool Suite) maintained within repository at ESRL/GSD/EMB for HIWPP]• Rapid Update Analysis would provide gridded truth for many fields

Top 3 System Performance Strengths• Flexible: deterministic & ensemble guidance, myriad of stats & metrics, portable• Content for all models, long records of stats, stored in database, retrievable • Accessible, locally at least, familiar to developers, interactivity

Top 3 System Performance Challenges• Separate complex systems across the agency & enterprise, no common repository• Access, reliability & user support is lacking, more user friendly, limitations remain• Lack of reliable truth fields and need for quality control of obs

5

System Evolution Over the Next 5 Years

Major forcing factors• Major dev efforts e.g. NGGPS, WOF require fast, flexible verif to make reapid progress• Need a shared & unified verification system that spans global-scale to storm-scale• Challenges of verifying/detecting features, svr wx, clouds, etc & their attributes

Science and development priorities• Unification & sharing of common codes for all required verification capabilities• Enhanced diagnostics & graphics including spatial distribution of errors & model difs• Enhancements: obs impact monitoring, octahedral & cube-sphere projections, sub

1km spacings, using more sat prod, non-PrepBUFR (e.g. Aeronet) and RapUpdAnal• Ens. PBS stat type to support reliability diagrams, Brier Score, decomp of Brier Score,

and Brier Skill Score against the sample climatology. What are you top challenges to evolving the system(s) to meet

stakeholder requirements?• Unification & sharing of common codes for all required verification capabilities• Resources: personnel, compute, storage, training, common computer workspace• Utilizing & QC of new data sources, and validating new techniques for verification

Potential opportunities for simplification going forward• Building a unified, fast, flexible, comprehensive verification system (incorporating

MET/METViewer and HIWPP/EMB elements / features) will GREATLY facilitate the most rapid skill improvement for all NCEP model systems [WPC adopting MET too]

• Partnering is more common e.g. Stony Brook R2O proj & hazards detection• Using scripting languages (e.g. python) and workflow managers (e.g. Roccoto)

6

Top 3 Things You Need From the UMAC

1. Push for INVESTMENT in and unification of verification across the enterprise: MET/METviewer versus MATS versus in-house written systems. EMC, NGGPS, WOF, FACETS really need this.