“Data Sources: Pulling it all together”

John Schneider

Global Systems Division

Earth System Research Laboratory

11/29/2012 1

AMS Board on Enterprise Communications

Improving Weather Forecasts

November 29, 2012

• Research regular concept

• Open Weather/Climate Service concept

• Best science and data to industry

• Needs for observation densities

• National Network of Networks (NNON)

• Incentives for sharing/participation for both

NOAA and private sector

• What elements should the Government vs.

private sector maintain

• High quality and lowest price

11/29/2012 2

An Approach to Integrate and Distribute Data

In 2001: – Need: Accurate and reliable data from sustained and

integrated earth observing systems.

• Improved data interoperability and usability

• common data management standards

• Enhanced access and use of environmental data through data storage and

access solutions, integration of systems, and long-term stewardship; and

• Reduced life cycle cost of observations through increased partnerships,

integration of systems leveraging available data, and reducing

unnecessarily duplicative capabilities.

– Opportunity: Internet ideal pathway for bringing in the

islands of observations. The approach:

• Create partnerships with other government agencies, the private sector, and

the public

• Provide a service that adds value to the data.

– Quality Control all data.

– Create a common output format that simplifies access.

– Provide a more dense observation infrastructure.

11/29/2012 3

• Meteorological community has access to 69,917 stations from

over 160 surface networks producing nearly 13 million

observations per day.

• Uniform data formats, observation units, and time stamps.

• Observational Quality Control (QC).

• Network-enabled distribution with server-side sub-setting.

• Authentication for proprietary data.

11/29/2012 4

NOAA

Surface Stations

NOAA and Partners

Surface Stations

The NOAA Operational System

– MADIS system being transitioned from Initial Operating

Capability (IOC) (above) to Final Operating Capability

(FOC).

• Primary and Backup systems hosted at NOAA Web Operation

Center (WOC) operationally maintained by NCEP.

• MADIS archive hosted at National Environmental Satellite, Data,

and Information Service (NESDIS).

11/29/2012 5

• The Meteorological Assimilation Data Ingest

System (MADIS)

– runs operationally at the National Weather Service

(NWS).

• Conduit for providing new and improved

observational data to NWS operations and greater

meteorological community:

– National Mesonet (NM)

• US Mesonet Consortium

– Global Science & Technology, Inc. (GST)

– WeatherFlow, Inc.

– University of Utah (UU) - MesoWest

– University of Delaware (UDEL) - DEOS

– Rutgers, The State University of New Jersey - NJWxNet

– North Carolina State University (NCSU) - ECONet

– Coastal Carolina University (CCU) – South Carolina Network

– GST’s Mobile Platform Environmental Data (MoPED).

– FAA 1 Minute ASOS/AWOS data.

• Pathway for Federal Highway Administration’s

Clarus system to move to operations.

11/29/2012 6

11/29/2012 7

• 1990’s – Isolated islands of observational networks.

• Stove pipes of information.

• Designed and built for specific operational needs.

– Internet that was created in 1969 was just starting to take hold as

a viable communications pathway for everyone. • 1990 World Wide Web protocols finished.

• 1991 first web page and first content based search engine (gopher) were created.

• 1993 first widely downloaded graphical web browser for the general public, Mosaic, was released, the

White House and United Nations come on line marking the start of the .gov and .org domains.

• 1998 Google (search engine) and Napstar (file sharing).

11/29/2012 8

• Extended and enhanced QC.

– Ability to add algorithms e.g. Clarus.

– Improved meta-data from NM and Clarus.

• Improved data and meta-data standards.

– NextGen

– Starfish Fungus Language (*FL)

– Sensor Model Language (SensorML)

• Faster and more access points to MADIS processed

data.

– Raw

– Different levels of QC.

– Complete QC.

• Improved displays (Google Earth).

– Combined data and meta-data.

– Cookies for saving user preferences.

– Archived/Real-time

11/29/2012 9