hmt-dtc task
DESCRIPTION
HMT-DTC Task. Ed Tollerud In collaboration with ESRL, RFC’s, WFO’s, and CDWR. DTC Goals and the HMT. Provide early step up for front-line forecasters vis-à-vis ensemble usage Demonstrate usefulness of prototype ensemble forecast systems; thereby - PowerPoint PPT PresentationTRANSCRIPT
Ed Tollerud
In collaboration with ESRL, RFC’s, WFO’s, and CDWR
HMT-DTC Task
DTC Goals and the HMTProvide early step up for front-line
forecasters vis-à-vis ensemble usage
Demonstrate usefulness of prototype ensemble forecast systems; thereby
provide long-range guidance for ensemble forecast systems
Enhance and motivate MET development
HMT-West ensemble QPF evaluation is current focus to attain these goals
Major accomplishments - past 6 monthsOnline WRF ensemble
verification in near-real-time:Regionalization of HMT domain30-day summariesObservational data options
Object-based (MODE/MET) verification of historical atmospheric rivers
Direct comparison between GFS and WRF ensemble mean
Online QPF verification for HMT-West
Equitable Threat Scores (ETS) for ensemble members
Forecasts initiated at 1200 UTC 1/17 2010
Ensemble mean (black) GFS (green)
30-day summary boxplots, ETS
January 2010 Large HMT domain Intensity
thresholds: .01,.1,.5,1,3 in
Planned activities - next 6 monthsRevise and expand winter exercise verification
demonstration websitePrepare and present preliminary results from
2009-2010 winter exercise verificationDevelop probabilistic capabilities for QPF in
the HMTContinue retrospective study of MET/MODE
applications to atmospheric river forecasts
Anticipated major accomplishments - next 6 months
Following features added to HMT-West winter verification exercise: 1-2 new additional baseline models (SREF,
NAM,…)Time series display of MODE attributes Displays of reliability and ROC diagrams6h gage accumulation options
Plans for AOP 2011Incorporate spatial verification statistics for
ensemblesImplement METviewer online displaysCollaborate with DET to apply MET-based
probabilistic verification techniquesDevelop procedures to assess gridded QPE
products and their impact on verificationExtend verification to additional
meteorological fieldsImplement prototype AR verification utility
Anticipated major accomplishments – AOP 2011Revised and expanded verification options
appropriate for HMT (West and East)Comprehensive suite of probabilistic
verification utilities applied to QPF for HMTObject-based verification techniques
implemented for AR forecastingReports and presentations from HMT-West
verification activities
Resource requirements for AOP 2011Staff Category FTE
Scientists 1.0Software Engineers 0.5
Students 0.1
Scientists Software Engineers
Ed Tollerud John Halley Gotway
Tara Jensen Paul OldenburgCIRES
associateRandy Bullock
RAL statistician Brent ErtzTressa Fowler Chris HarropBarb Brown
Non-salary CostTravel $7 K
Publications $2 K
Challenges (Science and Resources)Establishing synergistic roles of DET and DTC
vis-à-vis ensemble verificationDetermining potential use and advantages for
regional and episodic verificationDeveloping appropriate down-scaling methods for baselining and comparisonImproving forecaster access to results of evaluation (ALPS? AWIPS?)Validating long-range point precipitation forecast for Tillamook for September 2011
Tara Jensen
HWT-DTC Collaboration
Collaborating with:NOAA: SPC, NSSL, HPC, AWC, ESRL, EMCUniversities: OU/CAPSNCAR: MMMUCAR: COMET
HWT-DTC Collaboration Goals• Explore a different R2O Paradigm
• “O” is Forecasters, WFOs, and Prediction Centers• Evaluate convection allowing models on leading edge
of NWP• Gather robust dataset for ensemble and deterministic
studies• Datasets and algorithms leveraged by DET
• Demonstrate utility of MET-based objective evaluation in a forecast environment• Provide near real-time evaluation during Spring Experiment• Application of MET feeds back to MET development
• Facilitate faster R2O through focused evaluation of key topics identified during Spring Experiment• Perform retrospective studies using DTCs METviewer
Major accomplishments - past 6 months
Enormous expansion of near real-time evaluation capability Evaluation of 30 models during Spring Experiment
(CAPS ensemble+3 operational baselines) 10 Deterministic and 4 Ensemble products evaluated using
traditional and spatial verification methods. Three additional research models available for retrospective
studies
DTC staff participation in each week of Spring Experiment 2010
Manuscript accepted by AMS Weather and Forecasting.(Kain et. al, 2010 on Impact of radar assimilation on short-term forecasts)
2010 HWT Spring Experiment – 30 models
CAPS SSEF Ensemble PM Mean
CAPS 1 km Model
CAPS SSEF
ARW-CN (control w/
radar assimilation)
3 km HRRR
12km NAM
CAPS SSEF ARW-C0
(control w/o radar
assimilation)
• Assimilation continues to show greater skill over no-assimilation
• CAPS Ens. Mean generally hashigher skill than 1km deterministic and Operational baselines
Planned activities - next 6 monthsComplete re-run of
Spring Experiment evaluation
Add additional probabilistic measures(Brier Skill Score, ReliabilityDiagrams, and ROC Curves)
Perform retrospective evaluation of all members/products
Probabilistic Verification ofEnsemble Products (for a Single Run)Brier Score and decomposition
CAPS SimpleCAPS NeighborhoodSREF Simple
Anticipated major accomplishments - next 6 monthsAnalysis for presentations (Severe Local Storms and
AMS)Investigate physics perturbations on Accumulated Precip. and
ReflectivityEvaluate performance of probability products for QPFDetermine meaningful metrics from object-oriented
evaluation of Radar Echo Top
Report on impact of Radar Assimilation on 0-12 hr forecast
Targeted analysis using METviewer to facilitate 2011 Spring Experiment planning
Quality-checked and complete dataset available to collaborators for their own investigation
Plans for AOP 2011Contribute to R2O into SPC, HPC and AWC?
by providing objective evaluation for their Spring Experiment foci:SevereConvective Initiation/LightningQPF/Heavy RainEnsemble Product Evaluation
Perform retrospective studiesProvide feedback to researchers on ensemble
configuration and product generationLeverage DET Verification ModuleHelp develop link between DET and HWT
Anticipated major accomplishments – AOP 2011Near real-time objective evaluation for:
Simulated Satellite Lightning ProductQPF and Probabilistic (PQPF) fieldsReflectivity (if desired)
Evaluation products available within 30 min – 2 hours of valid time (rather than 6-18 hours)
DTC staff member participation during each week of Spring Experiment
Submit Journal article on QPF/PQPF evaluation
Resource requirements for AOP 2011Staff Category FTE
Scientists 0.54Software Engineers 0.31
Students 0.15
Scientists Software Engineers
Tara Jensen {System Engineer}
Michelle Harrold
Paul Oldenburg
Jamie Wolff* John Halley Gotway
Isidora Jankov* Brent ErnstTressa Fowler*
Lisa Coco
Non-salary CostTravel $9 K
Workshops $1 KPublications $1.5 K
Other (see below)
Compute Resources: HPC facility like NOAA-Jet system or NCAR Bluefire systemGoal is to use DET Verification Module.
* Spring Experiment participant
ChallengesHow to balance expectations with budget
HWT Collaboration started smallSuccess has garnered more requests and expectationBudget has not grown
How to serve the needs of additional testbedsAWC planning to start separate Aviation Weather Testbed
Indicated interest in DTC participation - How do we fund activity?HPC planning to continue using HWT facility
What if their evaluation needs expand?
Defining HWT-DTC Collaboration and DET interactionWhat is appropriate evolution of these complimentary
tasks?
Zoltan Toth
DTC Ensemble Testbed
Collaborators: NOAA/NCEP/EMC
DET website: http://www.dtcenter.org/det
Goals for setting up new DET task
Develop and maintain DET infrastructureSupport its use by community
Establish NCEP operational system as benchmark
Test and evaluate new community methodsTransition successful methods to NCEP and
other agenciesLink up with ensemble work in other
test-beds
Module 1: Configuration
Module 2: Initial
Perturbations
Module 4: Statistical
Post-Processing
External Input(HMT, HWT, HFIP,
etc)
Module 3: Model
Perturbations
Module 5: Product
Generation
Module 6: Verification
DET
MO
DU
LES
Major accomplishments - past 6 monthsInitial plans developed for
Overall architecture of DET infrastructureEach of 6 modulesTest and Evaluation
Initial tests for establishing basic capabilities for 2 modulesConfigurationInitial perturbations
Collaboration with other DTC tasks / projectsHMT – Joint plans for testing DET & HMT ensemble generationHWT – Joint plans for evaluation of 2011 HWT ensemble
OutreachOrganized DET Workshop and engaged with WRF Ensemble
WGActivities coordinated with NCEP/EMC via regular meetings
DET WorkshopOrganized by DTC with input from WRF Ensemble WG
Aug. 18-19, BoulderExcellent opportunity for connecting DET with
community50 plus participants from community & agencies
Total of 6 Breakout sessions focused onConfigurationPreliminary plans for testing
Extensive set of important recommendationsTo be used for updating DET plans
WRF Ensemble WG meeting, Aug 20Additional recommendations
Planned activities - next 6 monthsComplete planning for overall design and 6
modulesDevelop and test initial perturbation methods
Collaborate closely with NCEP/EMC for their next implementation
Complete protocol for T&EContinue coordination / collaboration with
HMT & HWTLeverage winter 2010/11 HMT ensemble for
DET testingPrepare for joint evaluation of 2011 HWT
ensemble
Anticipated major accomplishments - next 6 monthsDetailed plan for infrastructure – overall
design plus 6 modulesBasic capability for modules 1-2
Initial tests contributing to next NCEP implementation
In collaboration with NCEPProtocol for T&E & basic capability for
verification module (#6)Joint plans / activities with HMT & HWT
Testing of generation & verification of ensemble (HMT)
Evaluation plan of ensemble (HWT)Basic capability – Functionality tested /
supportedBenchmark – Methods used in
operations can be replicated
Plans for AOP 2011Benchmark for module 2
Contribute to next NCEP implementation; Replicate next NCEP configuration
T&E for module 2Use T&E protocol and verification module
Contribute to NCEP implementation for module 2 (FY11-12)Test improved method, contribute with algorithm & software
Basic capability for module 3Initial tests with NEMS compatible and WRF models
Contribute to design and evaluation of HMT & HWT ensemblesCollaborative effort
Anticipated major accomplishments – AOP 2011Benchmark for module 2
Milestone: Build initial perturbation module replicating next NCEP SREF implementation
T&E for module 2Milestone: Complete tests of new initial perturbation configuration
Contribute to NCEP implementation for module 2 (FY11-12)Milestone: Collaborate with NCEP scientists and contribute to
development of new initial perturbation methodBasic capability for module 3
Milestone: Demonstrate basic capabilities for model-related uncertainty (NEMS / WRF)
Contribute to design and evaluation of HMT & HWT ensemblesMilestones:
Test DET benchmark configuration for initial perturbation module in HMT ensemble Contribution to evaluation of HWT ensemble
Resource requirements for AOP 2011Staff Category FTE
Scientists 2.1Software Engineers 0.8
Scientists Software Engineers
Barbara Brown Paula McCaslinIsidora Jankov Linda Wharton
Tara Jensen Eugene MirvisEd Tollerud Ann Holmes
New Hire (DET Lead)
Non-salary CostTravel $8 K
Workshops $15 KPublications $2 K
ChallengesIdentify CPU resources for computer intensive DET work
Internal GSD & NCAR resources very limitedPrepare & submit proposals for external resources
Agencies and community desire quick setup of & extensive testing by DETMismatch between funding and expectationsHow can DTC build benchmarks faster?How can DTC engage other agencies / programs / projects?
HFIP, NUOPC, NEXTGEN, FAA, AFWA, etcContingency plan assuming additional funding
Accelerate integrated development of & community support for modules 4-6Basic capability for modules 4-5 – AOP11Benchmark for modules 4-6 – AOP12
DET TIMELINE – STARTING MARCH 2011AOP11 deliverables – Basic capabilities, benchmarks
Basic capability for module 3Benchmark + T&E for module 2Contribute to NCEP implementation for modules 2 (FY11-12)Contribute to design and evaluation of HMT & HWT ensembles
AOP12 deliverables – Benchmarks, T&EBenchmark + T&E for module 3Basic capability for modules 4-5T&E of community methods for module 2
AOP13 deliverablesBenchmark for modules 4 & 6Contribute to NCEP implementation for module 4 (FY13-14)T&E of community methods for all modulesUpdate benchmarks
Laurie Carson
DTC NEMS
Collaborators: NOAA/NCEP/EMC, NOAA/ESRL
DTC NEMS GoalsDefine community software goals for the
NEMS modeling framework and operational model configurations
Gain expertise in the NEMS framework, by participating in and assisting the on-going development, with a community support focus
Develop an O2R and R2O ops-concept that supports the needs of both the research and the operational user
Progress to-date: past 6 monthsExplore approaches and ideas to help define DTC NEMS
community software goals and timeline On-going: internal (DTC) and external (EMC, NCAR, GSD)
discussionsNEMS software status and near-term plans at EMC1-2 year plans including FIM, RapidRefresh, SREFTechnical expert meeting planned to focus on the research and operations
connections between WRF and NEMSDevelop DTC expertise in NEMS software
On-going: technical discussions with NEMS developers Code repositoryTask/Work plan for Software Engineer hire
Software Engineer hire to focus on NEMS development (at EMC)starting Oct 4, 2010
Technical Meeting: October 5, 2010
Planned activities - next 6 monthsDTC Software Engineer
Will reside at EMC to further facilitate the information exchange and DTC contributions
Add in-depth knowledge of NEMS to the DTCContribute to the development of NEMS softwareBegin work on a draft code management plan
Hold a technical information exchange meetingFall 2010: DTC, NOAA-EMC, NOAA-GSD, NCAR-MMMoverlap between WRF and NEMS with respect to
community supportmaintain a connection with the broad research communityuse this as a path-finder to explore R2O and O2R
mechanisms
Anticipated major accomplishments - next 6 monthsRefine work plan for DTC-NEMS software engineer
Add I/O layer capabilities to NEMS that will benefit research communities
Computing Platform Portability (systems, compilers)Inter-operability with other community softwareUsers Guides, sample datasets, and examples of
useTechnical meeting Oct 5, 2010 in Boulder
Focus on upcoming interactions between WRF and NEMS components
Plans for AOP 2011Strengthen the foundation of DTC
expertise with the NEMS software Continue to contribute to the development
process for NEMS, with a community user focus.
Refine DTC NEMS community software goals and timeline
Develop, with EMC, a NEMS code management plan to support community software and operational system goals
Anticipated major accomplishments – AOP 2011DTC NEMS code management planContinued Technical Interchange MeetingsDTC NEMS community software plan,
including a path from R2O and O2R
Resource requirements for AOP 2011Staff Category FTE
Scientists 0.1Software Engineers 0.75
Scientists Software Engineers
Jamie Wolff Eugene MirvisLaurie Carson
Non-salary CostTravel $12 K
Workshops $300
ChallengesEngagement of the NWP research community
is a critical component to making contributions to the operational NWP capabilities
NEMS is still under active development, so any decision to package a software release to the research community must be done with careful coordination between EMC and DTC
A fully supported community software package requires considerable resources from DTC. Is this the best use of DTC resources?