content, discovery, and accessibility enhancements to the ncar research data archive

Content, Discovery, and Accessibility Enhancements to

the NCAR Research Data Archive

Doug Schusterand

Steve WorleyNCAR

Topic Outline

Archive ContentData Discovery and Access

Archive ContentCommon Data Management Tools

Data acquisition Metadata extraction and mapping Data archival

Data Management Tools• Old System: Specialized Software to manage each data

input.• Inefficient• Difficult to Scale

RDAMetadataDatabase

Unidata Server

University Server

NWPServer Online Disk

Tape Storage

GCMDMetadata

Server

RDA Data Server

Specialized Software

Package 2

Specialized Software

Package 3

Specialized Software

Package 1

• New System: Common RDA tools that homogenize data management.• Efficient• Scalable

RDA Data Management

Common Tool Set

Archive ContentPopular Datasets

In situ and satellite based observations Gridded analysis and forecast model data High resolution re-analysis

Popular RDA Datasets

Platform ObservationsDataset Title Coverage Update

FrequencyNCEP GDAS observations (PREPBUFR) Global 1999 - Present Daily

NCEP GDAS observations (BUFR) Global 1999 - Present Daily

NCEP GDAS Satellite Radiance Data (BUFR) Global 2004 - Present Daily

Unidata IDD GTS based observations (NetCDF) Global 2002 - Present Daily

NCEP operational observations (ON-29 Format) Global 1975 - 2007 Fixed

International Comprehensive Ocean-Atmosphere Data Set (ICOADS)

Global 1662 – Present

Monthly

1662 2010Global Platform Observations

Popular RDA Datasets

Analysis and Forecast Model DataDataset Title Coverage Update

FrequencyThorpex Interactive Grand Global Ensemble

(TIGGE) Global 2006 - Present Hourly

Unidata IDD (GFS 0.5deg, RUC 20km, NAM 12km) Global and Regional 2002 - Present Daily

NCEP ETA/NAM (40km) North America 1995 - Present Monthly

ECMWF Operational Deterministic (1.25 x 1.25 Deg) Global 1985 - Present Bi-Yearly

NCEP GDAS Final Analysis (1x1 Deg) Global 1999 - Present Daily

NCEP OI Global SST (1x1 Deg) Global 1981 - Present Weekly

NOAA OI Global SST (0.25 x 0.25 Deg) Global 1981 - Present Monthly

Hadley Centre Global Sea Ice and SST Global 1850 - Present Monthly

1850 2010Analysis and Forecast Model Data

Popular RDA Datasets

High Resolution Re-AnalysisDataset Title Coverage Update Frequency

ERA-15 (N80 Gaussian) Global 1979 - 1993 Static Set

ERA-40 (T159) Global 1957 - 2002 Static Set

ERA-Interim (N128 Gaussian) Global 1989 - Present Yearly

1890 2010High Resolution Re-Analysis

JRA-25 (1.125 Deg Gaussian) Global 1979 – Present Yearly

NCEP/DOE (T62) Global 1979 - Present Static Set

NCEP/NCAR (T62) Global 1948 - Present Quarterly

NARR (32 x 32 km) North America 1979 - Present Quarterly

CFSR Global 1979 - Present TBD

NOAA-CIRES 20th Century Global 1890 – 2008 Static Set

Data Discovery and AccessData Discovery

Popular dataset listGoogle style searchDrill down style searchEnhanced content metadata

Data DiscoveryLog inPopular Dataset ListGoogle Style

SearchDrill Down Style

SearchEnhanced Content

Metadata

Data Discovery and Access

Data Access MethodsUser created “dynamic” file list

and download optionsServer default “static” file list and

download optionsDelayed mode request and

download optionsSpecialized project portals

Data AccessInternet Download Options Matrix

Data Group TitleDiscover and Access Online Data Files for Immediate Download (40

TB)

Online Data Example 1: User Created “Dynamic” File List and

Access Options

Data AccessExample 1: User Created “Dynamic” File List

Select Desired Parameter(s)Select Type of

ProductSelect Vertical LevelEnter Desired Date RangeUpdate the List

Data AccessExample 1: User Created “Dynamic” File List

Select Files for downloadVolume SummaryDownload as

Single Tar FileDownload Full Files

with Wget ScriptDownload Select

Parameters with cURL Script

Data AccessExample 2: Server Default “Static” File List

Online Data Example 2: Server Default Static File ListClick on Magnifying Glass Icon

to Access File Level Content Metadata

Internet Download Options Matrix

Data AccessInternet Download Options Matrix

Discover and Request for Offline Data Files to be Staged Online for Delayed

Mode Download (500 TB)Example 3: Request Access to

Offline Data Files

Data Access

Example 3: Request Access to Offline Data FilesEmail Notification when

Delayed Mode Request is Ready for Download

Data AccessInternet Download Options Matrix

Email a Question Regarding Data Access

Special Project Portals• TIGGE Portal

– THORPEX Interactive Global Grand Ensemble Archive (TIGGE)

– Supports Weather research leading to an integrated Global Interactive Forecast System• Integrated across multiple international NWP

Centers • 10 NWP Centers contributing to TIGGE

• Future– Model Validation Portal (under design)– Upper Air Observations Portal (under

development)

Special Project PortalsTIGGE Portal

http://dss.ucar.edu