verifying hydrologic forecasts in the u.s. national ... · 02/02/2007 verification workshop,...
TRANSCRIPT
02/02/2007 Verification Workshop, Reading, UK 1
Verifying Hydrologic Forecasts in the U.S. National Weather Service
Julie Demargne1,2, DJ Seo1,2, Limin Wu1,3, John Schaake1,4 and James Brown1,2
1Office of Hydrologic Development, NOAA/National Weather Service (NWS)
2University Corporation for Atmospheric Research3RS Information Systems, Inc.
4Consultant
02/02/2007 Verification Workshop, Reading, UK 2
Outline
• Forecast Verification at NWS: needs and vision • Hydrologic Products and Services• River Forecast Verification System• Ensemble Verification System (EVS)
Results for precipitation and streamflow ensemblesNew graphics and measures
• Closing Remarks
02/02/2007 Verification Workshop, Reading, UK 3
Forecast Verification at NWS
• Needs:1996: National Research Council stated verification of hydrologic forecasts was inadequate
2006: Board on Atmospheric Sciences and Climate recommended NWS to expand verification of its uncertainty products and make it easily available to all users in near real time
• Vision:Develop a comprehensive national system to verify hydrologic forecasts and guidance products which satisfy the needs of all usersImprove forecast services by analyzing sources of error and skill across the entire forecast process Provide easy access to enhanced river forecast verification data to improve our scientific and operational techniques and services
02/02/2007 Verification Workshop, Reading, UK 4
Hydrologic Products and Services
• Consistent probabilistic forecasts for all lead times & verification information
02/02/2007 Verification Workshop, Reading, UK 5
Hydrologic Products and Services
A
B
NWS-NDGD Gridded
Uncertainty Product
B Products for hydrometeorological inputs and hydrologic forecastsA
H
Least Likely
Forecast Legend
Likely H
Least Likely
Forecast Legend
Likely H
Least Likely
Forecast Legend
Likely
Most Likely
_Median Fcst
? ObservedStage
Flood Stage
x
River Flood Outlook Products
NWS-NDFD High-Resolution Gridded Water Resources Product Suite (WRPS)
NWS-NDFD High-Resolution Geospatial Water Resources Product Suite (WRPS)
• Examples of ensemble graphical products
02/02/2007 Verification Workshop, Reading, UK 6
Hydrologic Ensemble Prediction System• Goal: to improve reliability and skill and to accurately account for
uncertaintiesReal-time forecasting & hindcasting modes
Atmospheric Ensemble Pre-Processor (EPP)
Hydrological Models
(& Regulation)
Data Assimilator
Hydrology & Water Resources Ensemble Product Generator
Hydrologic Ensemble Processor
Weather & Climate Forecasts
Observations
VERIFICATIONProducts & Services
02/02/2007 Verification Workshop, Reading, UK 7
River Forecast Verification System
• Goals:Quantify quality of RFC forecasts and quality of forecast servicesMonitor forecast quality over time Monitor quality at various steps in the forecast processImprove forecast qualityAssist prioritization of forecast system enhancements
• Uses:OperationalExperimental/Research
• Customers:Hydrologic forecasters Scientists/ResearchersHydrologic program managersEmergency and water resources managersPublic
02/02/2007 Verification Workshop, Reading, UK 8
River Forecast Verification System• Components
Logistical verification Deterministic verificationProbabilistic verification
• Verification System CapabilitiesData Archiving (attributes for time, service, basin and events) Computing MetricsDisplaying Metrics (graphics and reports)Disseminating Metrics and Data (understand quality & usefulness of forecasts)Real Time Access to Metrics (understand errors in recent forecasts and over long term)Error Analysis (including hindcast experiments)Performance Measure Tracking (level of success)
02/02/2007 Verification Workshop, Reading, UK 9
Ensemble Verification System (EVS)
• A prototype application intended for systematic verification of all the components of the ensemble prediction system:
Verification of ensemble hindcasts/forecasts for inputs (precipitation, temperature) and outputs (streamflow)Supporting application: Ensemble Hindcaster for systematic hindcasting (re-forecasting) based on operational forecasting system
• EVS consists ofA suite of data processing and science algorithms that:
processes observed and hindcast/forecast datacalculates a suite of verification statistics
A suite of R scripts for graphical display of verification statistics and datascatter plotsdeterministic and probabilistic verification metrics
A Java user interface
02/02/2007 Verification Workshop, Reading, UK 10
EVS Results: Test Areas at the NWS River Forecast Centers (RFC)
Ensemble forecasting test basins
• Verification study for 5 test basins within ABRFC area:Ensemble hindcasts for precipitation and streamflow for 14 lead daysVerification period: 03/2003 - 08/2005 (890 events at 24-hr time step)
02/02/2007 Verification Workshop, Reading, UK 11
Hydrologic Ensemble Processor
EPPprecipitation ensembles
EPP-based streamflow ensembles
Climatology-based streamflow ensembles
EVS Results: Forecasts to be verified
Ensemble Pre-Processor
(EPP)
Climatological ensembles
Single-valueQPF
(days 1-2)
• Precipitation forecasts generated by Ensemble Pre-Processor:Single-value (i.e. deterministic) forecast has additional skill from human forecasters, particularly in Day 1Precipitation ensembles from NWP have significant biases in meanand spreadGoal of Ensemble Pre-Processor is to produce precipitation ensembles that are unbiased and that reflect additional skill
A practical, “observation-driven” approach for ensemble generation (see Schaake et al. 2007 in HESS)
• Streamflow forecasts generated from:EPP precipitation ensemblesClimatological ensembles (operational process)
02/02/2007 Verification Workshop, Reading, UK 12
EVS Results: Brier Score
BS = Reliability – Resolution + Uncertainty
Perfect score: BS=0
EPP precipitation
85th percentile
EPP-based streamflow (w/ sim)
85th percentile
• Brier Score (mean squared probability error):EPP precipitation ensembles EPP-based streamflow ensembles compared to simulated flows
to evaluate only impact of input error
02/02/2007 Verification Workshop, Reading, UK 13
EVS Results: Brier Skill Score• Brier Skill Score (w/ climatology as reference):
EPP precipitation ensembles
EPP-based streamflow ensembles compared to simulated flows
-> input error
EPP-based streamflow ensembles compared to observed flows
-> input and hydrologic errors
Climatology-based streamflow ensembles compared to simulated flows
-> benefits from EPP
BSS for different forecasts
85th percentile
Perfect score: BSS=1
02/02/2007 Verification Workshop, Reading, UK 14
EVS Results: Reliability Diagram
Histogram for day 1
Deviation from diagonal indicates conditional bias
Reliability Diagram(agreement between forecast probability and mean observed frequency)
EPP precipitation ensembles
EPP-based flow ensembles (w/ sim)Streamflow forecast verification with simulated flows to assess:- impact of input error - benefits of using Ensemble Pre-Processor
02/02/2007 Verification Workshop, Reading, UK 15
EVS Results: ROC
Perfect scores: HR = 1 and FAR = 0
Diagonal: no skill
Relative Operating Characteristic (discrimination)(with 10 probability thresholds)
Climatology-based flow ensembles
EPP-based flow ensembles (w/ sim)Streamflow forecast verification with simulated flows to assess:- impact of input error - benefits of using Ensemble Pre-Processor
02/02/2007 Verification Workshop, Reading, UK 16
EVS: New Verification GraphicsModified box-and-whisker plot: marginal distribution of
forecast error (xens - xobs) for each event
100 percent.
90 percent.80 percent.
50 percent.
20 percent.10 percent.
0 percent.
EPP precipitation ensembles (w/o zero observed)
Observed value (mm)
Fore
cast
err
or (m
m)
For 1 event
02/02/2007 Verification Workshop, Reading, UK 17
EVS: New Verification Measures
Error window (+/- %)
Pro
babi
lity
that
obs
erva
tion
falls
in e
rror
win
dow
Size of error window
Cumulative Talagrand Plot: probability that observation falls in error window around median of probabilistic forecast
Probability
40% of time, obs. should fall in window ±20%
“Underspread”
“Hit rate”= 89%
50 percent.Error
window xens
100 percent.
0 percent.
xobs
For 1 eventEPP precipitation ensembles (w/o zero observed)
02/02/2007 Verification Workshop, Reading, UK 18
EVS: New Verification Measures
% of ensemble members within error window (averaged over time)
Average Capture Rate Plot: probability that ensemble members fall in error window around observation, averaged over time (mean of Wilson (1999) score)
Fore
cast
err
or (m
m) Different
“acceptable" errors
Average capture rate50% of members within
± 2.5 mm on averagexobs
Probability of falling in
error window“Avg. Max Error”= 47 mm
Error window
For 1 eventEPP precipitation ensembles (w/o zero observed)
02/02/2007 Verification Workshop, Reading, UK 19
Closing Remarks
• Hydrologic ensemble verification is relatively newDevelop new or adapt existing verification metrics that are easily understandable and informative for decision making
• Hydrologic forecasts are subject to many different sources of uncertainty (forcing inputs, initial conditions, models, etc.)
Identify and quantify uncertainties at every step of forecasting system (with hindcasting capabilities)
• Closer interdisciplinary collaborations (e.g. HEPEX) will help maximize the utility of weather and climate forecasts in hydrology and water resources applications
Communicate both meteorological and hydrologic uncertainties to the users for better decisions
