A presentation for the 4A presentation for the 4thth COPS Workshop COPS WorkshopSeptember 25-26, 2006September 25-26, 2006Hohenheim, GermanyHohenheim, Germany

Raymond McCord Raymond McCord Oak Ridge National Laboratory*Oak Ridge National Laboratory*

Oak Ridge, Tennessee, USAOak Ridge, Tennessee, USA

Assisted by Dave TurnerAssisted by Dave TurnerUniversity of WisconsinUniversity of Wisconsin

Madison, Wisconsin, USAMadison, Wisconsin, USA

*Oak Ridge National Laboratory is operated by UT-Battelle, LLC, for the U.S. Department of Energy under contract DE-AC05-00OR22725*Oak Ridge National Laboratory is operated by UT-Battelle, LLC, for the U.S. Department of Energy under contract DE-AC05-00OR22725

Overview of Atmospheric Radiation Overview of Atmospheric Radiation Measurements (ARM) Data Management Measurements (ARM) Data Management

and Archiving in NetCDF formatsand Archiving in NetCDF formats

OverviewOverview• Data management

– Objectives – Policy

• ARM data and systems description– Systems overview– Data storage strategy

• About Data Files and Formats– Features– Header attributes– Data structure– Access and Analytical Tools

• ARM Data and Information Types• Beyond “the data file”

– Where are the metadata??– Web tour of www.arm.gov

• ARM Data Access– Overview of Archive– Demo of ARM Archive user interfaces (time allowing)

Quotes from RaymondQuotes from Raymond• “Storing data is EASY. Finding and using data later

is NOT…”– Data accessibility and usage, not storage, are the primary

metrics of an Archive• “Systematically and consistently organized data

does not occur without cost. Consider the results from previous science projects with no extra effort for data archiving.”

• “The natural tendency over time for data and information is chaos. Effort must be exerted to overcome this.”

• “Successfully managed data by projects may not be ready to be archived.”

• Scientific data systems must be designed to accommodate changes (content, access, users, etc.). This is noticeably different from business systems – the origin of most of our technology.

Data Management: ObjectivesData Management: Objectives

• ARM Objectives– Create a data product that is:

• Logically and structurally consistent through time• Capable of accommodating changes (scope, content, quality

information, etc.) • Accessible both “now” and in the future

– Develop and operate a data system that is:• Timely to develop and processes data in a timely manner• Modular for expansion and change• Can withstand external review (mostly scientific and quality issues)

• COPS Objectives– When possible create data products “like ARM”– When possible attain the same data management objectives as

ARM

ARM Data PolicyARM Data Policy

• Provide open data access:– To maximize exchange of data

• between collaborating programs• to be available for scientific objectives

– In a timely manner (known and minimal delays)– To data of “known and reasonable” quality– From routine instrument operations– With delayed and restricted access for experimental implementations

• Record data usage and users– Retrospective notifications of new quality information or reprocessed

data– Important for documenting “worth” of data to sponsoring organization– Required for “National User Facility” status

• Provides access to operational funding beyond research programs

Data SystemsData Systems

ARM Data Systems: OverviewARM Data Systems: Overview

GeneralScientific

Community(2100 users,

140 universities44 countries)

Southern Great Plains

North SlopeOf

Alaska

TropicalWesternPacific

AerialVehicles

ARMArchive70 TB

Data Mgt &Processing

Facility

ARMScientists

External- Model- Satellite- GIS (BNL)

(ORNL)

(Alaska)

(Manus, Nauru, Darwin)

(Oklahoma, USA)

(PNNL)

• Geography Dispersed• Enabled by Internet Technology• Continuous availability• Today - >2000 Different Data Streams• Availability/Quality/Meaning

25 GB/Day

MobileFacility

ARM Data Systems: DetailARM Data Systems: Detail

laptop

laptop

Data logger

Site data systems

ResearchUser system

Very Limited User Access

ARMDMFhourly

con

tin

uou

s

Shared disk

hour

lyExternal disk(shipped)

ARMArchive

As needed

daily

hourly

Research / Data Quality system

ARM Data Storage StrategyARM Data Storage Strategy

• ARM data are stored in Data Streams– A “data stream” is a series of files (daily) that have

similar contents and structure.• Files can be concatenated across time if needed.• Daily files are created as a convenience for processing,

review, transfer, and distribution.

• The same instruments at different locations create files with the same data stream structure.

• Automated QC flags are contained within the data files.

About Data Files and FormatsAbout Data Files and Formats

NetCDF File “features”NetCDF File “features”

• Processed ARM data files are stored in NetCDF format– Self-contained data documentation

• Header block• Data arrays

– Non-proprietary format (open source) – Efficient binary format– Directly accessible by application software (IDL, MATLAB)– Libraries available for data creation and access from your own

software • available for Fortran, C, C++, Perl, Java

• http://www.unidata.ucar.edu/software/netcdf/index.html

NetCDF File Structure (Header)NetCDF File Structure (Header)

• File-specific information – creation time, dimension values for arrays

• Data definition attributes– Data field names (varname)– Data field description (longname)– Data limits

• min, max – optional

– Measurement info • units, resolution, missing value code, etc. – optional??

• Global values (attributes)– Descriptive information that valid for a portion of the data stream

• Location name, reference for retrieval algorithm, long term calibration information, contact information, etc.

Examples of ARM Header InformationExamples of ARM Header Information

Online Demo Link Here

NetCDF File Structure (Data)NetCDF File Structure (Data)

• Data are stored in “array” records after the header.– ARM data are “dimensioned” by Time and sometimes Height

• Time recording is very important. • ARM uses base time + time offset and composite time

– Multi dimensional arrays are possible, but rare.

• Data fields are stored in the same order as defined in the header.– Data are accessible by “array number”– Avoid using this!!!

• Single and multiple dimension data arrays can occur in any order within a data stream.

NetCDF Data Access and Analysis NetCDF Data Access and Analysis

• Applications using NetCDF can:– Access data by filename / data field name– Concatenate similar files (e.g., from a time

series)– Merge of values based on similar dimension

values

• Links to NetCDF tools can be found at:– http://www.arm.gov/data/tools.stm

ARM Data/Information Structure ARM Data/Information Structure

Going to a “higher” view!!

ARM Data Types - overviewARM Data Types - overview

• Continuous data (stored offline, accessible by requests from user interface)– ARM collected data– Value added products– External data

• Special data (stored online, accessible from web interface)– Field Campaign (IOP) data– Beta data– PI generated data products

ARM Data Types – more detailARM Data Types – more detail

• ARM collected data– RAW data files

• Available upon request, but not accessible from User Interface• Minimal documentation; user beware• Wide variety of formats; many are binary

– Processed data files• Accessible from user interfaces• Common formats include NetCDF and HDF

• Value added products (VAPs) – Include one or more of the following

• Advanced algorithms• Multiple data inputs• Input from long-time periods

– ARM produces some VAPs to improve the quality of existing measurements. In addition, when more than one measurement is available, ARM also produces "best estimate" VAPs.

armarchive@ornl.govarmarchive@ornl.gov1-888-ARM-DATA1-888-ARM-DATA

Types of Quality InformationTypes of Quality Information

• Automated products – QC flags

• inserted in data files during processing

– QA flags– Summaries of flags (data color)

• Manual products– Data Quality Reports (DQRs)

• web accessible reports • delivered as html files after data requests • event driven and problem-based

– Mentor Instrument Reports• web accessible (http://www.db.arm.gov/IMMS/ )• Also linked to instrument web pages.

Beyond the Data File!!Beyond the Data File!!

• Overview of Information Structure– “Patience… Please… getting ready for a Web

Tour”

• You will benefit from our “logic”.• You will need our “content”.• We will need to know your “content”.• Your structural “logic” will also be helpful to us.• A “sneak attack” on Metadata Issues

????

VAPs

Guest

ARM information StructureARM information Structure

Sites

“Instruments” Data streams Measurements

Location, etc

Categories +

metadataDocumentation + Categories +

Metadata

Data stream

“Family” metadata

Tour of www.arm.govTour of www.arm.gov

Data streams Measurements

Instruments

What do you see now??

Data Access (user interfaces) Data Access (user interfaces)

How many doors are enough??

Accessing Data from the ArchiveAccessing Data from the Archive• User interface options

– Overall scheme of user interfaces– Logical view of interfaces

• More details and demo (time allowing)– ARM Data Browser– Web Shopping Cart– Catalog Interface– Thumbnail Browser– IOP Data Browser– Contact Us…..

• 1-888-ARM-DATA, armarchive@ornl.gov

• Continuous data distribution– “Standing Orders”

You are NOT alone...You are NOT alone...• 3 sites• 10’s facilities• 100’s data sources• 100’s data users• 1000’s measurement

types• 1,000,000’s data files• 1,000,000,000’s

measurements• 10,000,000,000,000’s

bytes

Request Statistics From ArchiveRequest Statistics From Archive

Archive Data FlowArchive Data Flow

0

400000

800000

1200000

1600000

2000000

2400000

Oct-95 Oct-96 Oct-97 Oct-98 Oct-99 Oct-00 Oct-01 Oct-02 Oct-03 Oct-04 Oct-05

files MB

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

Oct-95 Oct-96 Oct-97 Oct-98 Oct-99 Oct-00 Oct-01 Oct-02 Oct-03 Oct-04 Oct-05

MB in MB out

Comparison of User Interface OptionsComparison of User Interface OptionsInterface

nameAccessible

data“Shopping” approach

(armarchive@ornl.gov, 1-888-ARM-DATA)

ARM Data Browser

Routine ARM data

“I know what I want. Do you have it?”Searching with predefined selection criteria.

Catalog Interface

Routine ARM data

“I am not sure what I want. I need to see what you have available.”Browsing a hierarchy of availability summaries.

Thumbnail Browser

Most routine ARM data

“I will know what I want when I see it.”Searching with a combination of predefined selection criteria and visual review of data plots

Web Shopping Cart

Routine ARM data and some IOP data

“I need to read about what you have, then I will decide.”Discover areas of interest by browsing the ARM web documentation and collect items of interest.

IOP Data Browser

IOP, special, PI, and beta data

“I need to look in the odd parts bin.”Direct access to IOP data. Navigate /year/site/iop directory tree. Also use narrow Google search.

Overall Interface SchemeOverall Interface Scheme

Display detailed information(file list, DQRs, color map, QLs)

Display summary results from search(# files, # DQRs, # QLs)

Identify “data of interest”(answer questions)

Order files

You and the Archive You and the Archive (Simplified view)(Simplified view)

Archive web-basedUser Interface

Database

quer

y spe

cific

atio

ns

quer

y res

ults

User copy (FTP)

E-mail notification

MassStorageSystem

File list and tracking

FileRetrievalProcessor

Requested files

FTP host

Start

End

User Interface “Demo”User Interface “Demo”

use presentationGo to web interface

Display ThumbnailsDisplay Thumbnails

Thumbnail Browser – Catalog InterfaceThumbnail Browser – Catalog Interface

Thumbnail Page

IOP Data Browser – IOP ViewIOP Data Browser – IOP View

Click for access to Click for access to more data sub-more data sub-

directoriesdirectories

IOP Data Browser – Data SelectionIOP Data Browser – Data Selection

NotificationProcessor

Standing Order ProcessingStanding Order Processing

Data base

Email specifications to Archive

New File Processor

User copy (FTP)

E-mail notification

FTP hostftp.so.archive.arm.gov

NewData files

Temporarycopy

DeliveryDirectories

Order specifications

Questions? Comments?Questions? Comments?

Detailed Reference SlidesDetailed Reference Slides

Data access policy “goals” (1)Data access policy “goals” (1)

• Data exchange between ARM and COPS as open and complete as needed– (more comments on next slide)

• Provide online documentation about– Measurement technology– Installation and site information– Data structure– Basic QA review methods and results

• Generate data products in a “timely” manner – Predictable schedule for generation and access

• Retain complete and comprehensive records of data inventory, usage, and users.– In a searchable database

• Distribute to data users updated information for data quality and data revisions (reprocessing) as needed

Data access policy “goals” (2)Data access policy “goals” (2)

• Assume that fully open access has the best potential for overall scientific output– No cost for data exchange and access

• Protect “rights” of data generators– Provide initial opportunity for publication and evaluation

• Especially for data from “new” instruments.– Offer co-authorship or acknowledgement to instrument PI’s.

• Prevent premature access of data– Very early access only as needed for operational planning (forecasting)– Before initial QC evaluation is complete

• Recipients of data have unrestricted use.• Within an “access group” all requestors have equal access

– No favorites between groups (??)• Data file format (netCDF) and structure will match ARM when

possible (??)

ARM Archive SystemsARM Archive Systems

DMFsystem

ExternalData system Archive

Storage Processing

User interface

MassStorage System

daily

Metadata Database

As needed

Retrievalprocessing

FTP host

usersStanding

OrdersExternal disk

(shipped)

Radar Spectral data

IOPData system

usersuser

s ARM Webdocumentation

Logical Structure of ARM MetadataLogical Structure of ARM Metadata

MET

SKYRAD

InsturmentClass

description

Date range

Site / facilitylist

Inventories ofStored and

Retrieved filesData stream

Measurement metadata

InsturmentCode

description

zcc1metM1.b1

nim1metM1.b1

30met

1met

skyrad60s

skyrad20s

nim30metM1.b1

zcc30metM1.b1 DailyFiles

Dailyfiles

Dailyfiles

Dailyfiles

Storageprocessing

UserInterface

Meastype

WebInfo