![Page 1: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/1.jpg)
TIGGE Archive Access at NCAR
Steven WorleyDoug SchusterDave StepaniakHannah Wilcox
![Page 2: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/2.jpg)
Background on TIGGE
WMO World Weather Research Programme THORPEX
THe Observing system Research and Predictability EXperiment
THORPEX Interactive Global Grand Ensemble (TIGGE) Archive supports researchGrand Ensemble = multiple NWP centers ensembles are
combined (an ensemble of ensembles) 10 international NWP Centers contributing to TIGGE
![Page 3: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/3.jpg)
Background on TIGGE
Three mirrored archive centers NCAR ECMWF CMA{Shared System Development!}
Daily Data Flow Metrics245 GB1.6 Million gridded fields as separate data packets
![Page 4: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/4.jpg)
Data Receipt
Archive Centre
Current Data Provider
NCAR NCEP
CMC
UKMO
ECMWFMeteoFrance
JMAKMA
CMA
BoMCPTEC
IDD/LDM
HTTP
FTP
Unidata IDD/LDMInternet Data Distribution / Local Data ManagerCommodity internet application to send and receive data
NCDC
![Page 5: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/5.jpg)
Major ChallengesInsure data receipt, build complete archive
Collate data fields into different files typesExchange manifest files as part of IDD/LDM data
transmission between Archive centersVerify send, receiveAutomated resend requests for missing fields
Harvest and hold metadata in MySQL DB’s Identify location of every field in file setUpdated often Critical for users interface and background data
processing
![Page 6: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/6.jpg)
Major Challenges Limited online storage – 4 TB, ≅ 2.0 weeks
temporal coverageFull archive on NCAR Mass Storage System
User registration and metrics requiredAccept data policy; for research and education only48 hour delay from forecast initialization time
![Page 7: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/7.jpg)
Major Challenges Access system must accurately display what
data are available as users make selectionsDriven by multi-center research (Grand Ensemble)E.g., want to experiment with NCEP and JMA
forecasts for 500 hPa temperatures Maximum common data is determine by JMA
12Z forecast initialization and 9-day forecast
![Page 8: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/8.jpg)
Differences between centers
![Page 9: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/9.jpg)
User access demonstration
Animation, what you will seeMultiple centers
(ECMWF, UKMO, NCEP, CMA, CMC, KMA)Fields/Parameters
(Geopotential Height, 2m Temperature)Levels
(500 hPa, Single Level)Spatial and temporal ranges
(Global, 3-days, 12Z initialization, 48 hour forecasts)Regridding to common spatial resolution
(1.5°)Output format
(netCDF)
![Page 10: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/10.jpg)
Sample Data Request for an Event
![Page 11: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/11.jpg)
![Page 12: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/12.jpg)
![Page 13: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/13.jpg)
![Page 14: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/14.jpg)
Retrieve Completed Subset
![Page 15: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/15.jpg)
![Page 16: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/16.jpg)
Subset Request Animation
![Page 17: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/17.jpg)
Gustav/Hannah Animation
![Page 18: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/18.jpg)
Lessons Learned
Manifest files and automated resend are critical for a complete archive
The impact of different contributions from the NWP centers across archive cannot be under estimated
There are important design considerations to insure prompt browser interactions Caching data from the DB
![Page 19: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/19.jpg)
Lessons Learned
Computational resource requirements ramp up quickly with multi-dimensional problemsD’s, center, ensemble member, parameter,
forecast length, etc. Archive file structure choices greatly impact
subsetting abilityTIGGE currently based on synoptic orderTime-series by parameter could be better?
![Page 20: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/20.jpg)
End
http://tigge.ucar.edu/
![Page 21: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/21.jpg)
Future
Consider adding storage resources to keep a longer period of data online
Integrate more computation resources to reduce time to fill requests
Create a way to re-stage MSS data to online for internet users
![Page 22: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/22.jpg)
Outline
Brief Background on TIGGE Archive Data Receipt Challenges Access Lessons Learn
![Page 23: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/23.jpg)
User registration and metrics required
Conditions for useResearch and education only48-hour delay on availability
![Page 24: TIGGE Archive Access at NCAR Steven Worley Doug Schuster Dave Stepaniak Hannah Wilcox](https://reader036.vdocument.in/reader036/viewer/2022062401/5a4d1b5c7f8b9ab0599ab4a9/html5/thumbnails/24.jpg)
Limited online storage – 5 TB, ≅ 2.5 weeks
1-2 TB additional storage for user request preparation
Supplement AccessDirect Archive File download, though interface or
Research Data Archive (RDA)All data (200 TB) available through RDA from
NCAR Mass Storage System TC data?