bridging the gap in operational hydrologic data …rto) transition plans for the hydrologic ensemble...
TRANSCRIPT
1 Data Assimilation Workshop, Delft Nov 1-3, 2010
Bridging the gap in operational
hydrologic data assimilation -
Challenges and a way forward
D.-J. Seo1, Yuqiong Liu2,3, Haksu Lee2,4, Victor Koren2, Jiarui
Dong5,6, Michael Ek5, Pedro Restrepo2
1The University of Texas at Arlington, Arlington, TX, USA
2NWS/Office of Hydrologic Development, Silver Spring, MD, USA 3Riverside Technology, Inc., Fort Collins, CO, USA
4University Corporation for Atmospheric Research, Boulder, CO, USA 5NWS/National Centers for Environmental Prediction, Camp Springs, MD, USA
6I. M. Systems Group, Inc., Rockville, MD, USA
2 Data Assimilation Workshop, Delft Nov 1-3, 2010
Acknowledgments APRFC
And many collaborators in the research and operational communities
3 Data Assimilation Workshop, Delft Nov 1-3, 2010
In this presentation
• Context of this presentation
• DA in operational hydrology in NWS
– Past, present
• Lessons learned
• Needs
• Way forward
– Strategy, opportunities
4 Data Assimilation Workshop, Delft Nov 1-3, 2010
Community Hydrologic Prediction
System (CHPS)
• Modular software to enhance collaboration and accelerate R2O
• Extension of the Flood Early Warning System (FEWS) architecture:
– Incorporates NWS models with models from FEWS, U.S. Army
Corps of Engineers (ACE), and academia
Flexible, open modeling architecture linking program elements
Implementation Status: AWIPS-II compatible prototype
hardware and software for all RFCs
Conducting parallel operations at 4
RFCs, remaining by early 2011
Retire legacy system in early 2012 FEWS
NWS
Models
ACE
Models
Other
Models
FEWS
Models
From Carter (2010)
5 Data Assimilation Workshop, Delft Nov 1-3, 2010
CHPS and Hydrologic Data Assimilation (DA)
From http://www.weather.gov/oh/hrl/chps/index.html
6 Data Assimilation Workshop, Delft Nov 1-3, 2010
DA planning with the RFCs (2008)
• Develop Phase 2 R&D and research-to-operations
(RTO) transition plans for the Hydrologic Ensemble
Forecast System (HEFS)
• Identify ensemble and DA capabilities that may address
new and existing service needs (particularly in light of
recent experiences and emerging issues)
– Implementable in CHPS
– Steer enhancement/evolution of FEWS
– Strengthen collaborations with the external research
community
7 Data Assimilation Workshop, Delft Nov 1-3, 2010
(Extra) Motivation
• One can only forecast based on all available information
– There are/will be more sources of information
– There are/will be significant additions to the forecast process
• They will require additional attention and intervention, and
expertise and judgment of human forecasters
• Making dynamically and statistically coherent/consistent sense
out of them is and will be a large challenge
• It is (past) time to rebalance what human forecasters (should) do and
what computers (should) do
– Forecaster role: Master analyzer, interpreter, judge and translator of
all available hydrologic, hydrometeorological and
hydroclimatological information
• In CHPS, we have a “miss-it-or-lose-it” window of opportunity to do this
rebalancing toward realizing the integrated water science and services
vision
8 Data Assimilation Workshop, Delft Nov 1-3, 2010
A brief history of DA in NWS
9 Data Assimilation Workshop, Delft Nov 1-3, 2010
Blending (Adjust-Q)
• Practiced ever since river forecasting began
• Merges the observed hydrograph with the simulated
• Does not correct the underlying cause of the discrepancy
• Is objective and easy to implement
• Cannot correct for certain common sources of errors
such as erroneous precipitation reports or timing errors
10 Data Assimilation Workshop, Delft Nov 1-3, 2010
CHAT (Computed Hydrograph
Adjustment Technique)
• Sittner, W. T. and Krouse, K. M. (1979) Improvement of hydrologic
simulation by utilizing observed discharge as an indirect input
(Computed Hydrograph Adjustment Technique-CHAT). NOAA Tech.
Memo. NWS HYDRO-38, US Dept of Commerce, Silver Spring.
• Applicable to runoff events not involving snow and to headwater basin
outflow hydrographs
• Adjusts the precipitation input and modifies the shape of the unit
hydrograph until the simulation is in satisfactory agreement with the
discharge observation
• Utilizes a unique objective function that models, to some degree, the
thought processes which a human forecaster uses in judging the
seriousness of a disagreement between the rising limb of a simulated
hydrograph and the discharge observations
– Unlike conventional optimizing techniques, does not seek to
minimize the objective function, but rather, attempts to reduce it to
an acceptable value
11 Data Assimilation Workshop, Delft Nov 1-3, 2010
Assimilation of snow course data
• Carroll, T. R. (1979) A procedure to incorporate snow course data
into the National Weather Service River Forecast System.
Proceedings, Modeling of Snow Cover Runoff (AGU, AMS, Corps of
Engineers and NWS, Hanover, New Hampshire, September 1978),
pp. 351-358.
• Incorporates snow course data into the NWSRFS
• Weights the simulated and observed water equivalent values
• Intended primarily to reduce monthly volume errors
• Is of particular value where the precipitation data do not accurately
represent the snow accumulation at higher elevations
– Basins where precipitation stations are located primarily at
elevations lower than the snow courses
12 Data Assimilation Workshop, Delft Nov 1-3, 2010
State updating via Kalman filter
• Kitanidis, P. K. and Bras, R. L. (1978) Real time forecasting of river flows.
Ralph M. Parsons Laboratory for Water Resources and Hydrodynamics,
Dept of Civil Engineering, MIT, TR235.
• Uses SAC (re-)written in continuous state-space form with certain
modifications
– Linearized using various techniques and then integrated to obtain the
state transition matrix for a discrete time Kalman filter
• An automatic procedure to estimate the model noise covariance matrix and
input noise covariance
• A generalized likelihood ratio test to detect input errors
• Limitations
– Considered only SAC with a crude procedure to time-distribute channel
inflow and route to the gauging station
– Considered the model and input noise covariance to be constant
– The procedure used to identify input errors did not consider the pure
timing error
13 Data Assimilation Workshop, Delft Nov 1-3, 2010
SS-SAC (State-Space Sacramento)
• NWSRFS Operation No. 22
• Georgakakos, K. P., and J. A. Sperflage, 1995: Hydrologic Forecast
System - HFS: A user’s manual. HRC Tech. Note 1, Hydrologic
Research Center, San Diego, CA, 17 pp.
• Performs real time updating of SAC and channel-routing model
storage elements from observations of discharge, and provides
discharge forecast-error variance
• Uses user-supplied degree-of-belief estimates for the variance of
SAC parameters and of forecast evapotranspiration demand and
precipitation or rain-plus-melt input to SAC
• The channel model component is a conceptual kinematic channel
routing model consisting of a series of nonlinear or linear reservoirs
– In the latter case, the unit hydrograph coordinates may be used
to estimate the channel-routing model parameters
14 Data Assimilation Workshop, Delft Nov 1-3, 2010
ASSIM (Assimilator Operation)
• NWSRFS Operation No. ?
• Koren, V. and Schaake, J. (1993). Nile Technical Note #147:
Updating Algorithm and Program.
• A soil moisture updating technique which modifies current soil
moisture states
• Uses the iterative Rosenbrock Optimization Technique
(Rosenbrock, 1960) to estimate the current 'optimal' soil moisture
states
• Initial soil moisture states and input precipitation are varied over the
run period and an objective function that includes the difference
between observed and simulated flows, precipitation adjustments
and state adjustments, is minimized
15 Data Assimilation Workshop, Delft Nov 1-3, 2010
SEUS (Snow Estimation & Updating System)
• McManamon, A., R. K. Hartman, and R. Hills, "Implementation of the
Snow Estimation and Updating System (SEUS) in the Clearwater
River Basin, Idaho", Proceedings: 63rd Annual Western Snow
Conference, pp. 56-65, Sparks, NV., April 1995.
• Consists of three components – calibration, operational, updating
– The updating component
• Modifies the existing snow water equivalent states of the
conceptual snow model within NWSRFS based on the
weighted contributions of the simulated model snow states
and the estimates of the snow states developed using the
snow observations
• The weighting for each estimate reflects the relative
uncertainty of the estimate due to such factors as model bias
and measurement error
16 Data Assimilation Workshop, Delft Nov 1-3, 2010
Snow Updating System
• Snow updating user manual, Version 2.00.02 Updated 2003-09-26,
Riverside Technology, Inc.
• An extension to NWSRFS and provides tools to update the snow conditions
• Consists of a graphical user interface (GUI) and batch programs, which are
based on a Principal Components Analysis (PCA) program
– The batch system can process all or a subset of the basins in a system
– The GUI simplifies running a subset of the system and also provides
displays to view the data and results
• Initially consisted only of the batch snow update software, which was
implemented by Riverside Technology, inc. for the Bonneville Power
Administration. The GUI was subsequently added, using BPA support and
technologies developed by RTi
• The PCA software was developed by Dave Garen of the USDA
17 Data Assimilation Workshop, Delft Nov 1-3, 2010
Recent prototype development
• 2DVAR
– State updating for SAC-UHG
– Seo et al. (2003)
– Implemented in the Site Specific Hydrologic Prediction (SSHP)
system, used at a number of WFOs and RFCs
• 4DVAR
– State updating of gridded SAC and kinematic-wave routing in
Research Distributed Hydrologic Model (RDHM)
– Lee et al. (submitted to AWR)
• 1DVAR
– Real-time updating of 3-parameter Muskingum routing
– Under integration in CHPS via OpenDA
18 Data Assimilation Workshop, Delft Nov 1-3, 2010
2DVAR - Comparative evaluation
19 Data Assimilation Workshop, Delft Nov 1-3, 2010
FA VAR-aided SAC-UHG forecast
FC MODa-aided SAC-UHG forecast
aRun-time modification by human forecaster
Seo, D.-J., L. Cajina, R. Corby and T. Howieson, 2009:
Automatic State Updating for Operational Streamflow
Forecasting via Variational Data Assimilation, 367,
Journal of Hydrology, 255-275.
20 Data Assimilation Workshop, Delft Nov 1-3, 2010
From “When NWSRFS, SEUS, and Snow
Intersect.” Vernon C. Bissell, Western Snow
Conference, Bend, Oregon, 1996.
From Snow updating user manual (2003)
21 Data Assimilation Workshop, Delft Nov 1-3, 2010
Operation #RFCs
ADD/SUB 13
ADJUST-H 1
ADJUST-Q 13
ADJUST-T 2
API-CONT 1
API-HFD 1
API-MKC 1
BASEFLOW 4
CHANGE-T 13
CHANLOSS 11
CLEAR-TS 13
CONS_USE 3
DELTA-TS 3
DWOPER 3
FFG 10
FLDWAV 6
GLACIER 1
LAG/K 13
LAY-COEF 1
LIST-FTW 2
LOOKUP 11
LOOKUP3 6
MEAN-Q 13
MERGE-TS 12
MULT/DIV 4
MUSKROUT 3
NOMSNG 9
PLOT-TS 8
PLOT-TUL 13
RES-SNGL 11
RSNWELEV 4
SAC-SMA 12
SARROUTE 1
RES-J 7
SET-TS 9
SNOW-17 10
SSARRESV 1
SS-SAC 1
STAGE-Q 13
STAGEREV 1
TATUM 2
TIDEREV 2
UNIT-HG 13
WEIGH-TS 10
Reality NWSRFS Operations in use (as of October 2007)
• Snow Updating System used at NWRFC
• 2DVAR, implemented in the Site-Specific Prediction System (SSHP), is used by various WFOs and RFCs
22 Data Assimilation Workshop, Delft Nov 1-3, 2010
2DVAR - Lessons learned
• Use the same, operational models (soil moisture accounting, snow, routing, etc.)
– Model physics and parameters must be the same and completely transferable
• Allow forecaster control/intervention
– To reflect any prior or additional information that the forecaster may have
– Restart (warm or cold) may be necessary if the model deviates from the real world
• Provide, and effectively present, model-dynamical information that explains the DA results
• Clearly demonstrate the value of DA through objective comparative verification
– In the context of the end-to-end forecast process
– Relative to the current operational practice
• Training
23 Data Assimilation Workshop, Delft Nov 1-3, 2010
From MBRFC’s presentation in XEFS Phase 2 R&D and RTO Planning (2008)
• Levee failures
Accounting for timing errors
due to spatially non-uniform
distribution of rainfall and/or
varying rainfall intensity
24 Data Assimilation Workshop, Delft Nov 1-3, 2010
From MBRFC’s presentation in XEFS Phase 2 R&D and RTO Planning (2008)
25 Data Assimilation Workshop, Delft Nov 1-3, 2010
For forecaster control
• For forecaster control of the end-to-end DA process, versatile,
informatics-based graphical user interface would be
necessary that allows, e.g., displays of with- and without-DA
results over multiple time periods for pattern identification.
• It is possible that, at times, the forecaster may have to restart
the DA process, going back to some user-specified time and
negating any changes thereafter. The DA tools should be
flexible enough to allow such forecaster-controlled warm
restarts.
From NWS/OHD Strategic Science Plan (2010)
26 Data Assimilation Workshop, Delft Nov 1-3, 2010
Science and technology
challenges • End-to-end DA
– A suite of integrated DA tools for forcings,
hydrologic models and hydraulics models
– Decomposition, control
• Synergistically linking land DA and DA for
operational hydrologic forecasting
27 Data Assimilation Workshop, Delft Nov 1-3, 2010
Closing thoughts
• Development, implementation and Institutionalization of forecaster-
supervised automatic DA in operational hydrology will require:
• Collective efforts from both the research and the operational
communities
• Cross-cutting community planning and action to capitalize on recent
advances and to seize the window of opportunity
• We, the cross-cutting community, need:
• A science roadmap to
− Lay out clear and scientifically-sound pathways to operationally-
viable practical solutions
− Identify and articulate specific scientific and technological challenges
therein
• Community-wide tools to
− Make R&D and RTO transition feasible
− Improve cost-effectiveness
− Foster cross-cutting collaborations
28 Data Assimilation Workshop, Delft Nov 1-3, 2010
Adapted from Integrated Water Science Plan (NWS 2004)
What kind of tools?
Airborne
Satellite
Radar
In-Situ
physiographic
New
physiographic
data
New
observing
systems
NOAA
NASA USGS
USDA
DOD
Operational data feeds
baseline developmental stream
Once validated, new data are
fed into operational stream
EPA
DOE
Operational Data
Assimilation
OTHERS
Experimental Data
Assimilation
DA Testbed
29 Data Assimilation Workshop, Delft Nov 1-3, 2010
TS
Parameters
States
CHPS-OpenDA
Model Adapter OpenDA
MODELS: SNOW-17, SACSMA UNITHG, Lag/K, …
CHPS/FEWS
DATABASE: Forecast Observation
What kind of tools? (cont.)
Community Contribution
Models DA Tools
VAR EnKF, EnKS MLEF, PF
Thank you
For more information on the
presentation, contact:
Hydrologic Ensemble Prediction System
Ensemble Pre-Processor
Parametric Uncertainty Processor
Data Assimilator
Ensemble Post-Processor
Hydrology & Water Resources Ensemble Product Generator
Hydrology & Water Resources Models
Hydrologic Ensemble Processor
QPF, QTFQPE, QTE, Soil Moisture
Streamflow
Improved
accuracy,
Reliable
uncertainty
estimates,
Benefit-cost
effectiveness
maximized
From NWS/OHD Strategic Science Plan (2010)
Hydrologic Ensemble Forecast System (HEFS)
Vision for Ensemble and Data Assimilation in Hydrologic Forecast Operations
Modeling & DA
From Snow updating user manual (2003)
From Snow updating user manual (2003)
Snow models
Soil moisture accounting
models
Hydrologic routing
models
Hydraulic routing models
reservoir, etc., models
In-situ snow water
equivalent (SWE)
In-situ soil moisture
(SM)
Streamflow or stage
Snowmelt
MODIS-derived snow cover
MODIS-derived cloud coverPrecipitation
Potential evap. (PE)
Runoff
Flow
River flow or stage
Flow
Atmospheric forcing
AMSR-derived SM1
AMSR-derived SWE1
MODIS-derived surface
temperature
1 pending assessment
SNODAS SWE
Satellite altimetry
From NWS/OHD Strategic Science Plan (2010)
NWS Operational DA Strategy
From Sorenson (2002)
From Sorenson (2002)
From Sorenson (2002)
The Red River Flood of 1997
• East Grand Forks city engineer Gary Sanders reasoned that predicting a crest at 49.0 ft in 1997 was putting the flood in the same class as the 1979 flood, which crested at 48.8 ft with a peak flow of 82,000 cfs
• The peak flow in 1997 reached 137,000 cfs
– "Missing by five ft may not sound like much, but when you talk about flow, the National Weather Service missed by almost 100 percent.“ (Sanders)
DA strategy for operational
hydrologic forecasting • Decompose
into smaller ones such that:
− The suboptimal solutions from the decomposed problems are
close to the optimal solution from the full-blown problem
− the resulting DA process is forecaster-controllable
nmnmnn
m
m
n V
V
V
X
X
X
HHH
HHH
HHH
Z
Z
Z
.
.
.
.
..
.
.
..
..
.
.
2
1
2
1
21
22221
11211
2
1
40
Multi-Model Soil Moisture Percentile for May 2008
D4 D3 D2 D1
Drought D0 < 20%
(a): http://www.cpc.ncep.noaa.gov/products/Drought/Drought_index/Drought_index.shtml
(a) NARR
(b) & (c): http://ldas.gsfc.nasa.gov/drought
(b) Noah
(c) Mosaic
D4 D3 D2 D1
Commonly dry regions: Southeast, Southern Plains, North Dakota, California
Snow Water Equivalent (a) 2008-06-01
http://www.nohrsc.nws.gov/nsa
(b) 2008-05-01
Snow melt providing water
over west mountain region.
Pacific Northwest: heavy
streamflow might due to
snow melt
(c)
42
STREAMFLOW FORECAST EXAMPLE 12-DAY LEAD-TIME (APRIL 1, 2006)
Ensemble mean
similar to analysis
– Error ~10% of flow
Ensemble spread
comparable to error
in ensemble mean
Analysis (NLDAS) Ensemble Mean
Error=Ens. Mean - analysis Ensemble Spread
From Hou et al.
2009
Strategy for bridging/coupling land data assimilation and hydrologic assimilation
NWS’s Integrated Water Science Vision
Coupled ocean/atmospheric/
land-surface model (global)
Coupled atmospheric/land-surface
model (regional)
Interface with Estuary Models,
Water Quality Models
WEATHER &
CLIMATE FOCUS
WATER FOCUS
Observing Systems
Common Hydrologic
Land-Surface Modeling System
(Uncoupled)
Research Labs
Land Surface Hydrology & Water Resources Modeling
From Integrated Water Science Plan (NWS 2004)
1___
1___
1___
1__
1___
1___
1___
1___
1__
*
stageriverV
moistsoilV
evappotenV
precipV
moistsoilX
stageriverZ
moistsoilZ
evappotenZ
precipZ
2___
2___
2___
2__
2___
2___
2___
2___
2__
*
stageriverV
moistsoilV
evappotenV
precipV
moistsoilX
stageriverZ
moistsoilZ
evappotenZ
precipZ
*
*
*
*
*
2___
1_____
2___
1___
moisturesoilV
moisturesoilVmoistsoilX
moistsoilX
moistsoilX
2___
1___
2___
1___
2___
1___
2__
1__
__
2___
1___
2___
1___
2___
1___
2__
1__
*
stageriverV
stageriverV
moistsoilV
moistsoilV
evappotenV
evappotenV
precipV
precipV
moistsoilX
stageriverZ
stageriverZ
moistsoilZ
moistsoilZ
evappotenZ
evappotenZ
precipZ
precipZ
Strategy for bridging/coupling land data assimilation and hydrologic assimilation (cont.)
Predicting Floods to Droughts
In Your Neighborhood
River channel
Developed
Shrubs/Grass
Agriculture
Wetlands
Forecast Point
Forecast Basin
River Services (600 miles per forecast point)
Water Resource Services (6 square mile forecast basins)
River Conditions Soil Conditions From Carter (2006)
Feb 3, 2010
Distributed Modeling System Simulated water balance model and channel routing states over the Arkansas River basin
before and after a storm; cumulated rainfall are shown at the top
From Koren
et al. 2004
Water Issues: Too Much, Too Little, Poor Quality
• New strategic plans for NOAA and NWS recognize decision-makers in all water management sectors need:
– Higher resolution information in space and time
– Quantification of uncertainty to manage risk
• Water resource challenges are significant and getting bigger:
– Population growth and economic development are stressing water supplies and increasing vulnerability
– A changing climate is impacting water availability and quality
– Socio-economic risks of floods and droughts are escalating
• Growing needs for water resource forecasts:
– Soil moisture for agriculture and forest management
– Low flow for maintaining water supply
– Water temperature and salinity forecasts for fisheries management and healthy ecosystems
From Carter (2010)