NCEP-EMC Global Modeling Roadmap Verification Project

Environmental Modeling Center

National Centers for Environmental Prediction

Monday, August 29, 2011

2

Summary of 08/29/2011 Meeting

Metric Specifics: • AC, Bias, RMSE etc: almost ready to apply for all model; needs to filter out missing cases from all models before making

maps.• Precipitation Skill Scores over CONUS: Fanglin is working on replacing current global verification program with Ying

Lin’s package. FIM has started to accumulate precipitation with a 6-hour bucket. NMMB needs to change its 12-hour bucket to the 6-hour bucket.

• Near-Surface Forecast Verification Statistics: Helin and Fanglin will work together to include it in the standard verification package. Need to explore the possibility of adding RMSE and BIAS etc to surface verification stats.

• Hurricane Track and Intensity Errors: Both GFS and FIM are running hurricane tracker programs and producing standard atcf files. NMMB needs either to add a tracker program to run with the model integration or to save high-resolution data to run an offline tracker program.

• Fit to Observations: There are three options: current global Fit-to-obs tool provided by Suru Saha, grid-to-obs tool developed in the EMC meso-scale branch, and another tool used by FIM. It is yet to be decided which one to use. Stan will provide more details bout the FIM tool. The current grid-to-obs tool is designed for verifying operational forecasts. Perry will put together an user-friendly grid-to-obs package that can be easily applied for verifying parallel experiments.

• 2-D Maps: Comparison of Actual Fields: Glenn and Fanglin will use their tools to make these comparison as needed.• Cloud properties and Radiative Fluxes: This will be considered in a later stage. Getting observations for quasi real-time

experiments is difficult. Need to decide which observations to use and what variables to be included for verification. • Synthesized Metrics: Glenn will run his Lazy-Man’s index as needed. Using the UKMO Index is presently not feasible.

Participants: Fanglin Yang, Jordan Alpert, Mark Iredell, Zavisa Janjic, Shrinivas Moorthi, Perry Shafran, Helin Wei, Glenn White, Ying Lin, and Stand Benjamin.

Fanglin gave a presentation detailing potential verification metrics to be included. Discussions were focused on model output, feasibility of certain verifications, and priorities.

Stan noted that 1) FIM is running a tracker program to produce hurricane track and intensity statistics, 2) FIM has added a 6-hour bucket to precipitation accumulation, and 3) FIM has added data at 20 hPa and 10 hPa to their standard output, 4) FIM is adding time-averaged radiative fluxes in its output.

Zavisa noted that 1) NMMB is not running any tracker program, 2) NMMB has two versions, one running with NMMB physics, and the other with GFS physics, 3) NMMB will change precipitation bucket from 12 hours to 6 hours.

We will have another meeting in a month or so.

3

IntroductionCandidate global NWP models that will be tested as members of a multi-model

ensemble at NCEP EMC in the future will be first tested for deterministic forecast at full resolution against the operational GFS. The candidate models are initialized with the operational GFS/GDAS analysis. A standard NCEP global modeling verification package will be put together for evaluating and comparing the forecast skills of these models.

In the past few years, a so-called VSDB-based verification package has been

developed at the NCEP/EMC for model evaluation. This package has been used by model developers at a few operational centers for evaluating the skills of operational global weather forecast models and for comparing operational and experimental forecast models. It includes verification metrics such as anomaly correlation, bias, root-mean squared error, variance, Murphy’s skill score, precipitation threat skill score, fits to rawinsonde observations, and hurricane track and intensity errors and so on.

We propose to use this VSDB-based verification package as a prototype for setting up a Standard NCEP Global Modeling Verification System. Existing components will be expanded and updated to allow it to ingest forecast data in different formats from different models. New components will be added to include more verification metrics.

4

Current Candidate Models• EMC GFS: http://www.emc.ncep.noaa.gov/GFS/

– T574 (~27km), 64 layers, spectral dynamical core, sigma-p vertical coordinate; Runs four times per day including both analysis and forecast; Forecast extends up to 384 hours.

– Output includes single-layer flux-type files (flx) at the 1152x576 Gaussian grid, and vertical profiles (pgb) on 37 standard isobaric layers extending from 1000 hPa to 1 hPa and at multiple horizontal resolutions (0.5, 1.0, and 2.5 degrees). GRIB1 format.

• ESRL FIM: http://fim.noaa.gov/– ~30km, 60 layers, Flow-following Finite-volume Icosahedral Model, isentropic-sigma

hybrid vertical coordinate; Runs with GFS physics; Forecasts initialized with GFS/GDAS analyses; Includes 00Z and 12Z cycles; Forecast extends up to 240 hours.

– Output: flux-type variables and vertical profiles are saved in the same file. 0.5-deg horizontal resolution, 41 vertical isobaric layers with a 25-hPa interval extending from 1000 hPa to 25 hPa, with 20hPa and 10hPa added; GRIB2 format. A 6-hour bucket has been added to precipitation accumulation.

• EMC NMMB: http://www.emc.ncep.noaa.gov/mmb/– Non-hydrostatic Multi-scale Model on the B grid; 1024 x 741 x 64, Runs with either

NMMB physics or GFS physics; Forecasts initialized with GFS/GDAS analyses; Includes 00Z and 12Z cycles; Forecast extends up to 216 hours.

– Output: similar to GFS

5

Verification Metrics

I) Anomaly Correlation, BIAS and RMSE etc

II) 2-D Maps: Comparison of Actual Fields

III) Precipitation Skill Scores over CONUS

IV) Near-Surface Forecast Verification Statistics

V) Fit to Rawinsonde Observations

VI) Hurricane Track and Intensity Errors

VII) Cloud properties and Radiative Fluxes

VIII) Synthesized Metrics

Color code: green – ready to go; blue – needs some work; brown – not clearly; red -- ??

6

I) Anomaly Correlation, Bias and RMSE etc

• Tool: The VSDB-based Verification Package developed at EMC has already been applied for verifying GFS, FIM and NMMB forecasts. It computes and saves verification stats as partial sums in VSDB database. GrADS is used to make graphics.

• All forecasts will be verified against GFS analyses, and independently against ECMWF analyses as well.

• Verifying Variable: Z, T, U, V, vector Wind, and SLP.

• Verifying Regions: NH (20N-80N), SH (20S-80S), Tropics (20S-20N), Pacific North America (20N-75N, 40W-180W).

• Statistics: Anomaly correlation, Bias, RMSE, Murphy’s MSE Skill Score, RMSE from Mean Difference, RMSE from Pattern Variation, Ratio of Standard Deviation etc

• Input data requirement: in GRIB1 or GRIB2 format; at any horizontal resolution no coarser than 2.5 degree; on isobaric layers including at least the layers of 1000, 925, 850, 700, 500, 400, 300, 250, 200, 150, 100, 50, 20, and 10 hPa.

• To-do list: verify every 6 hours instead of every 24 hours; remove missing cases to unify sample numbers on die-off curves.

Example: http://www.emc.ncep.noaa.gov/gmb/wx24fy/vsdb_glopara/pre13j/

7

II) 2D Maps: Comparison of Actual Fields

• Tool: Glenn White has a set of GrADS scripts that can be used to make zonal mean plots and lat-lon plots to compare forecasts from different runs. Fanglin Yang also has a set of GrADS scripts that can be used to compare all surface variables between different runs and certain variables between forecasts and observations.

• Data requirement: any type of data in GRIB format is acceptable. Ideally, data should be on lat-lon grid with a resolution no coarser than 1 degree. Vertical profiles should extend from the surface to about 1 hPa.

• This comparison has been proved to be useful for uncovering and debugging gross model errors.

Example: http://www.emc.ncep.noaa.gov/gmb/wx24fy/vsdb_glopara/pre13j/

8

III) Precipitation Skill Scores over CONUS

• For the GFS, Equitable Threat Score and BIAS score over the CONUS are computed during model integration using a legacy program Ying and Yuejian provided. The GrADS script wrote by Fanglin Yang reads the scores, computes Monte Carlo statistical significance and makes plots.

• Input data: needs 6-hourly accumulated precip saved at a resolution no coarser than 0.5 degrees; and the observed 24-hour accumulated precip over the CONUS, which is provided by Sid Katz at CPC/NCEP.

• Issues: FIM has added a 6-hour bucket for precip. NMMB uses a 12-hour bucket. The Zhu program needs to be modified to ingest NMMB data.

• Alternative Option: Ying Lin in the EMC Mesoscale Branch has a QPF verification package that is capable of handling precipitation with any accumulation bucket. It also use a newer dataset of CONUS gauge precip observation provided by NCEP-CPC. It saves precip scale scores in VSDB format. See http://www.emc.ncep.noaa.gov/mmb/yzhu/documents/precip_verif_user_guide.htm for the detail. Fanglin has started to learn and import Ying’s package for verifying GFS parallel experiments.

9

IV) Near-Surface Forecast Verification Statistics

• EMC Mesoscale Branch has been running routinely a near-surface verification package. Helin Wei has used it for verifying GFS global forecasts as well.

• Verifying variable: 2-meter air temperature and relative humidity, 10-meter wind speed

• Required input data : in GRIB1 or GRIB2 format; 3-hourly at any horizontal resolution no coarser than 2.5 degree. Observed data will be pulled from NDAS.

•

• Verifying Regions and forecast hours: CONUS: East half, West half, and some sub-regions. Only first 96-hour forecast will be verified.

• To-do List: The package can only be run offline currently. The

goal is to see if it can be merged into Fanglin’s vsdb-based verification system.

• Reference: http://www.emc.ncep.noaa.gov/mmb/research/nearsfc/nearsfc.verf.html

10

V) Fit to Rawinsonde Observations

• For the GFS, a fit-to-obs tool provided by Suru Saha has been used to compute the fits of analysis and forecasts up to 48 hours to rawinsonde observations during model integrations. An offline graphic package is then used to make plots.

• Required input data (?):

• Verifying Regions: NH, SH, Tropics, North America, Europe, Asian, the Glole.

• Verifying Variables: wind vector, temperature, geopotential height, relative humidity.

• • To-do List: Suru provided an offline version of the fit-to-obs tool

which Suru has been using for comparing ECMWF and GFS forecasts. Need to figure out how to run it offline to read FIM and NMMB data, to compute fit-to-obs stats, and to make graphics. It is yet to be decided what type of FIM and NMMB data to use.

• Alternative Option: EMC Mesoscale Branch has been running a Grid-To-Obs verification tool for verifying model forecasts against rao-obs. Needs to explore the possibility of unifying the tools between the global and mesoscale branches.

11

VI) Hurricane Track and Intensity Errors

• For the GFS, hurricane track and intensity stats are computed on the fly during model integration using a global_tracker.sh prpgram. A graphic tool built at EMC based on Tim Marchok’s early work is used to make track and intensity plots.

• FIM is running a tracker as well and generates actf files.

• Issues: 1) NMMB is not running any tracker program. 2) need to add statistical significance test.

• Guang-Ping Luo at EMC Mesoscale Branch developed a Cyclone Verification system that tracks cyclones in both the tropics and mid-latitudes. We may consider to include this verification as well.

12

VII) Cloud properties and Radiative Fluxes

• At ECM Global Modeling Branch, the validation of model forecasts of cloud properties and radiative fluxes against observations has not advanced as fast as did in the climate modeling community. Certain ISCCP and CLVAR-x products have been used for validating GFS forecasts. Jesse Meng at EMC has been collecting and processing SURFRAD data for land surface data assimilation and verification.

• Verifying variables to be considered: cloud cover (total, low, middle, high), total precipitable water, cloud liquid and ice water path, cloud effective radius, cloud optical thickness, surface and TOA longwave and shortwave fluxes, cloud radaitve forcing, cloud heating profiles etc.

• To-do list: – 1) need dedicated effort to collect and process most recent ground

and satellite observations since EMC parallel experiments are often targeted for the immediate past seasons. Some of the most commonly referred cloud and radiative products include ARM/DOE, SURFRAD, CLAVR-x, CloudSat, ISCCP, GCIP SRB etc.

– 2) may consider to use Cloud Simulator (http://cfmip.metoffice.com/ISCCP.html) for validating cloud forecasts.

13

Surface Radiation Verification -- Jesse Meng

• Observed Data: SURFRAD surface observationGOES satellite observation

Validated Variables: surface SW fluxsurface LW fluxsurface skin temperature

SURFRAD site locations

Q: can this tool be adopted for this project? Do we need to verify sfc radiation at this stage?

14

Glenn White --- Lazy man’s index

• --Count days significantly better or significantly worse using VDSB output

• --Use anomaly correlations, rms errors, murphy skill score (normalized rms error),• And pattern correlations

• --Heights, temperatures, vector winds

• --heights count only outside tropics

• --vector winds count only week 1

• --tropics count only week 1

• --At least a net 10% of days must be significantly better or significantly worse for it• To be significant

• --Count separately week 1 and week 2, NH, tropics and SH, troposphere (1000-200)• And stratosphere (100-10)

• Fit to obs—use numbers on time series plots—rawinsondes and aircraft

VIII) Synthesized Metrics

UK Index: needs persistence fcst scores. Still more work

NH

prd12q3h prd12q3i

SH

prd12q3h prd12q3i

1000 day 1-7 Days 1-7 Days 1-7 Days 2-5 Days 2-5

Days 8-14 Days 8-14 Days 8-14 Days 8, 9 Days 8, 13

700 days 1-7 Days 1-6 Days 1-6 Days 3-5 Days 3-5

Days 8-14

500 days 1-7 Days 1-6 Days 1-7 Days 3-5 Days 3-5

Days 8-14

250 days 1-7 Days 1-6 Days 1-7 Days 3-5 Days 3-5

Days 8-14 Day 14

Days 1-7 25-0/28 27-0/28 13-0/28 13-0/28Days 8-14 7-1/28 7-0/28 2-0/28 1-1/28

Anomaly correlations heightVerif. June 18-Aug 2, 2011

Forecast days significantly better or significantly worseThan operational GFS

Glenn White --- Lazy man’s index , sample result

16

All Models Running under NEMS

• In the future, all models will be running under the NEMS framework. The same post-processing module will be used to generate NEMSIO output which is on standard isobaric layers in GRIB2 format.

• Currently the aforementioned verification tools can only read GRIB1 data. These tools need to be updated to read GRIB2 data soon or later.

18

Comments – Steve Lord

1) Unification of verification system has been needed for years.  This is a good opportunity!!Go for it, but make it an affordable project so the model comparison is not delayed.

2) I prefer the Ying Lin precip. verification.  It is a WMO standard package.

3) The Marchok tracker is also a standard.  Should work with any set of GRIB files.

4) Clouds and Radiation are nice to haves, but can be applied later, since initial focusis on dynamics.  Simple difference maps can be helpful in determining the impact ofdifferent dynamics.

5) Why can't all models output GRIB2?  Can all diagnostic packages read GRIB2?

6) Need to unify grid-2-obs packages.  MesoBranch grid-2-obs should be sufficientfor the current purpose.  A special grid-2-obs that works on native grids should be usedfor some other purposes such as diagnosis for data assimilation.  In this latter case,calculations of (obs-ges) and (anal-ges) should be done exactly as in the data assimilationcodes.  This has been an issue for the past 20 years and no one has incorporated thisoption into the standard grid-2-obs package.  Long overdue.

7) near surface verification plan has little information.  This must be filled out by land group asap.

Please provide web sites where comparison statistics are being calculated.

19

Comments – Geoff DiMego

• Perry - My goal for you on this committee is to lobby on behalf of using established verifcations and not reinventing the wheel.  They don't have to use our packages exclusively, but you know what our packages can do, so you can help fend off any "silly ideas" they might come up with.

• For sanity check reasons, I'm forwarding this plan to Binbin so he can get you feedback on what he's done and is planning to do with respect to the grid-to-grid code which does grid-to-grid-ensemble package which (I think & hope) merges Yuejian's and Jun's probability verification packages into one.

• I'm also including Ying Lin since precipitation verification is on the list.

• Also, you might mention there is Guang-Ping Lou's cyclone verification package that might be considered as a supplement to the hurricane track & intensity.

20

Comments – Glenn White

• I think it is important to have an independent evaluation of the runs by someone not too involved in the various modeling efforts being evaluated.  In my not so humble opinion, the way I evaluate parallel runs is one way to evaluate the scores.  It does not produce one final number but does review all the fields and summarizes objectively the results of all the statistics.  The independent evaluation needs to be listened to but not necessarily agreed to.  Let the evaluator present his results and then let the debate begin. 

It is fine to evaluate the various runs against an external analysis (ECMWF for instance) but it should be remembered that in operations we do evaluate the system against its own analysis.Fit to rawinsondes is great, but it should be remembered that that excludes most of the tropics.I have come to regard the fit to rawinsondes of the analysis and first guess as a tunable parameter whose most appropriate values are determined by how well longer range forecasts do.Fit to other obs besides rwinsondes is needed as well.

• Assessing statistical significance is always a problem.  Fanglin assumes each day is an independent sample (except for precip).  We do not have measures of statistical significance for hurricanes.

• I do think assuming each day is independent is not a bad start.  There was a article by Bob Livezey 30 years ago on evaluating the signficance of regional scores that I think may be relevant. (Livezey, R.E., and W. Y. Chen, 1983: Statistical field significance and its determination by Monte Carlo techniques.  Mon. We. Rev., 111, 46-59.

21

Comments – Perry Shafran

• Our current grid-to-obs VSDB-based package verifies the model versus all sorts of observations - not just radiosondes, but also to all the ADPUPA obs, surface METARs, ship/buoy obs, aircraft, and satellite.  I think that it is an excellent tool that can be used to do nearly whatever you want.  I have used this in the past to verify global data, even on global subdomains (e.g. Europe, Asia, etc).  We have been using it for over 10 years now in the meso branch (we used it regularly in the North American Regional Reanalysis for verification).

• Keith Brill's FVS (Forecasting Verification System) handles the ingest of the VSDB data and does have the ability to create all sorts of statistical values.  One such example can be seen on page 9 of Fanglin's ppt - in the lower right hand corner, there is a plot that shows a nice diurnal cycle of data there - one such plot that we look at all the time.  That is just one of a large number of statistical plots that FVS is capable of producing.

• There is another, better web-based GUI that is currently under development for handling the VSDB files in a MySQL database format.   Currently it is working for only a small number of models but one of our goals in the next year or so is expanding it to include more models.

• Grid-to-obs is but one of the statistical packages we use in our branch centered around the VSDB format.  We have Ying's excellent precipitation verification system, which apparently you have been using an older version for some time.  Binbin's grid-to-grid is an excellent code that can verify model data against any grid analysis - its own analysis or another of your choosing.  Also, Guang Ping's tracker verification is an excellent tool to verify storm tracks, both speed and position.  All of them in unison makes a great and versatile verification tool.

• What our verification does NOT do at present is a horizontal plot of statistics on a map.  All our verification statistics are grid-averaged, and we lose a bit in the grid averaging.  We do have verification for a large number of pre-defined sub-domains, although I have added the ability to produce verification on a user-defined grid, given the corner lat/lons.  That is an area where you might want to stick to the processing that you already use, though that would be verification produced via a different code.  Another area where we might not be as sufficient is in the area of confidence limits - though Fanglin's use of our VSDB files to produce excellent graphs with confidence limits has been a great help to us.

• I would be happy to hear the reservations that global branch folk have regarding the VSDB verification tools.  We have for some time been hoping to unify EMC's verification around the VSDB format, and we thank Fanglin for his efforts in this area.  His contributions have been very valuable, and I look forward to helping the group develop a system suitable for your use.

22

Comments – Stan Benjamin

•

• slide 2 - FIM has 64 levels, output for 20 and 10 hPa being added (can do others as needed)

• slide 5 - We should verify SLP only where terrain < ~500 m.

• slide 7 - 6h precip buckets now available for FIM• slide 9 - We need to ensure WMO compliance

on raob verification• slide 10 - track info available for FIM (key

component of HFIP)

23

FVS instruction –Helin Wei

• the working machine is cirrus put the line below into your .cshrc source /u/wx32kb/vsdb/scripts/for_fvs

(1)create vsdb data from pgbf files /global/save/wx20hw/vsdb/prepfits/scripts.retro.3hr runpfit.sh date dir_vsdb dir_pgbf date: in YYYYMMDDHH format dir_vsdb: working directory to store the vsdb files dir_pgbf: directory in which pgbf files are located the sample script: job1.sh if the pgbf files are only available every 6-hr, use /global/save/wx20hw/vsdb/prepfits/scripts.retro.6hr

(2)generate fvs plots /global/noscrub/wx20hw/FVS_3R/

(a)the first step is to modify those files ending with sh2 and ctl2(*.sh2 and *.ctl) The sample set of these files are used to created fvs plots for two experimental cases (case name: PRC and PRJ) compared with observation replacedate.sh and replacename.sh are used to put the right starting and ending dates, case names in *.sh2 and *.ctl

(b)the second step is to run fvs.sh The script will assume the vsdb files are located in the directory dir_work/CASENAME dir_work is the directory where fvs.sh is located CASENAME: prc and prj in my sample the vsdb file name has to be in the format:casename_yyyymmdd.vsdb, if it is not, moven.sh is used to change the file name. the casename inside the vsdb files has to be consistent, so replacename.sh is used if not. the plots in ps format will created under dir_work/PLT after fvs.sh

Participants

Fanglin Yang, Jordan Alpert, Mark Iredell, Zavisa Janjic, Shrinivas Moorthi, Suranjana Saha, Perry Shafran, Helin Wei, Glenn White, Yuejian Zhu

Environmental Modeling Center, National Centers for Environmental Prediction

Stan Benjamin, John Brown, Michael Fiorino

Global Systems Division, NOAA Earth System Research Laboratory