![Page 1: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/1.jpg)
Content, Discovery, and Accessibility Enhancements to
the NCAR Research Data Archive
Doug Schusterand
Steve WorleyNCAR
![Page 2: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/2.jpg)
Topic Outline
Archive ContentData Discovery and Access
![Page 3: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/3.jpg)
Archive ContentCommon Data Management Tools
Data acquisition Metadata extraction and mapping Data archival
![Page 4: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/4.jpg)
Data Management Tools• Old System: Specialized Software to manage each data
input.• Inefficient• Difficult to Scale
RDAMetadataDatabase
Unidata Server
University Server
NWPServer Online Disk
Tape Storage
GCMDMetadata
Server
RDA Data Server
Specialized Software
Package 2
Specialized Software
Package 3
Specialized Software
Package 1
• New System: Common RDA tools that homogenize data management.• Efficient• Scalable
RDA Data Management
Common Tool Set
![Page 5: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/5.jpg)
Archive ContentPopular Datasets
In situ and satellite based observations Gridded analysis and forecast model data High resolution re-analysis
![Page 6: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/6.jpg)
Popular RDA Datasets
Platform ObservationsDataset Title Coverage Update
FrequencyNCEP GDAS observations (PREPBUFR) Global 1999 - Present Daily
NCEP GDAS observations (BUFR) Global 1999 - Present Daily
NCEP GDAS Satellite Radiance Data (BUFR) Global 2004 - Present Daily
Unidata IDD GTS based observations (NetCDF) Global 2002 - Present Daily
NCEP operational observations (ON-29 Format) Global 1975 - 2007 Fixed
International Comprehensive Ocean-Atmosphere Data Set (ICOADS)
Global 1662 – Present
Monthly
1662 2010Global Platform Observations
![Page 7: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/7.jpg)
Popular RDA Datasets
Analysis and Forecast Model DataDataset Title Coverage Update
FrequencyThorpex Interactive Grand Global Ensemble
(TIGGE) Global 2006 - Present Hourly
Unidata IDD (GFS 0.5deg, RUC 20km, NAM 12km) Global and Regional 2002 - Present Daily
NCEP ETA/NAM (40km) North America 1995 - Present Monthly
ECMWF Operational Deterministic (1.25 x 1.25 Deg) Global 1985 - Present Bi-Yearly
NCEP GDAS Final Analysis (1x1 Deg) Global 1999 - Present Daily
NCEP OI Global SST (1x1 Deg) Global 1981 - Present Weekly
NOAA OI Global SST (0.25 x 0.25 Deg) Global 1981 - Present Monthly
Hadley Centre Global Sea Ice and SST Global 1850 - Present Monthly
1850 2010Analysis and Forecast Model Data
![Page 8: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/8.jpg)
Popular RDA Datasets
High Resolution Re-AnalysisDataset Title Coverage Update Frequency
ERA-15 (N80 Gaussian) Global 1979 - 1993 Static Set
ERA-40 (T159) Global 1957 - 2002 Static Set
ERA-Interim (N128 Gaussian) Global 1989 - Present Yearly
1890 2010High Resolution Re-Analysis
JRA-25 (1.125 Deg Gaussian) Global 1979 – Present Yearly
NCEP/DOE (T62) Global 1979 - Present Static Set
NCEP/NCAR (T62) Global 1948 - Present Quarterly
NARR (32 x 32 km) North America 1979 - Present Quarterly
CFSR Global 1979 - Present TBD
NOAA-CIRES 20th Century Global 1890 – 2008 Static Set
![Page 9: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/9.jpg)
Data Discovery and AccessData Discovery
Popular dataset listGoogle style searchDrill down style searchEnhanced content metadata
![Page 10: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/10.jpg)
Data DiscoveryLog inPopular Dataset ListGoogle Style
SearchDrill Down Style
SearchEnhanced Content
Metadata
![Page 11: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/11.jpg)
Data Discovery and Access
Data Access MethodsUser created “dynamic” file list
and download optionsServer default “static” file list and
download optionsDelayed mode request and
download optionsSpecialized project portals
![Page 12: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/12.jpg)
Data AccessInternet Download Options Matrix
Data Group TitleDiscover and Access Online Data Files for Immediate Download (40
TB)
Online Data Example 1: User Created “Dynamic” File List and
Access Options
![Page 13: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/13.jpg)
Data AccessExample 1: User Created “Dynamic” File List
Select Desired Parameter(s)Select Type of
ProductSelect Vertical LevelEnter Desired Date RangeUpdate the List
![Page 14: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/14.jpg)
Data AccessExample 1: User Created “Dynamic” File List
Select Files for downloadVolume SummaryDownload as
Single Tar FileDownload Full Files
with Wget ScriptDownload Select
Parameters with cURL Script
![Page 15: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/15.jpg)
Data AccessExample 2: Server Default “Static” File List
Online Data Example 2: Server Default Static File ListClick on Magnifying Glass Icon
to Access File Level Content Metadata
Internet Download Options Matrix
![Page 16: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/16.jpg)
Data AccessInternet Download Options Matrix
Discover and Request for Offline Data Files to be Staged Online for Delayed
Mode Download (500 TB)Example 3: Request Access to
Offline Data Files
![Page 17: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/17.jpg)
Data Access
Example 3: Request Access to Offline Data FilesEmail Notification when
Delayed Mode Request is Ready for Download
![Page 18: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/18.jpg)
Data AccessInternet Download Options Matrix
Email a Question Regarding Data Access
![Page 19: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/19.jpg)
Special Project Portals• TIGGE Portal
– THORPEX Interactive Global Grand Ensemble Archive (TIGGE)
– Supports Weather research leading to an integrated Global Interactive Forecast System• Integrated across multiple international NWP
Centers • 10 NWP Centers contributing to TIGGE
• Future– Model Validation Portal (under design)– Upper Air Observations Portal (under
development)
![Page 20: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/20.jpg)
Special Project PortalsTIGGE Portal
![Page 21: Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive](https://reader033.vdocument.in/reader033/viewer/2022051420/56815e37550346895dcc9b18/html5/thumbnails/21.jpg)
http://dss.ucar.edu