noaa nextgen wx information database (widb) it architecture overview system of systems workshop

NWS Office of Science and TechnologySystems Engineering Center

(Skjei Telecom)

Mike Asmussen

1

Outline• Introduction• Requirements Development

– Requirements approach– Key documents– Requirements summary

• Generalized IT architecture– Assumptions– Architecture efforts to date– Wx Products– FAA IT Architecture– NWS / NOAA IT Architecture Summary

– Next Steps

2

Key Concepts• NWS / NOAA architecture will follow a System

of Systems approach• No one entity is building THE CUBE • It will be a federation of network-enabled

services (some existing services, and some yet to be deployed)

• Each service will be “registered” in a federated registry/repository, which • Will provide descriptive information about each

available service (via published metadata) and,• Will point any interested service user to the

appropriate service endpoint to allow service access4

Requirements Approach• Requirements Challenges

• Most available requirements are high level (not IT requirements)• No single document serves as the “definitive” requirements source• Some requirements not clearly characterized as:

• IOC/MOC/FOC• Single Authoritative Source (SAS) / non-SAS

• Approach• Surveyed numerous JPDO and NOAA documents• Determined relevant IT related requirements (as well as derived IT

requirements)• Categorized requirements• Created table of requirements

• Where possible, assigned timeframe of deployment and whether requirement is SAS-specific

• Identified Weak Areas• Only limited IT performance and security requirements have been developed

to date• IOC Cube contents are still under consideration (required use cases still

being examined)

6

Key Documents

7

Document Name Version Date SourceConcept of Operations for the Next Generation Air Transportation System V2.0 6/13/2007 JPDO

NextGen Network-Enabled Weather IT CONOPS 3.2 8/20/2008 NCAR, MITLL, NOAA/GSD

NextGen ATS Enterprise Architecture V2.0 6/22/2007 JPDO

Four-Dimensional Weather Functional Requirements for NextGen Air Traffic Management 0.1 1/18/2008

JPDO Functional Rqmts Study Team

Weather Concept of Operations V1.0 5/13/2006JPDO Weather Integrated Product Team

NextGen Weather Plan 0.6 3/20/2009 JPDOList of IOC and FOC products that NWS has committed to provide for NextGen

Final Performance Requirements (iFR) First Working Draft Wrapper - 4-D Weather Data Cube SAS Draft 2/11/2009 JDPONextGen Weather Information Database - Information Technology Needs (Draft SON) Draft 3/13/2009 OSTConcept of Operations and Operational Requirements - WIDB for the NextGen 07-042 5/4/2009

Office of Climate, Water and Weather Services

Definition of 4-D SAS 6/17/2009NEWP presentation by JPDO Wx Policy Team2

ATM Wx Integration PlanDraft V0.7 4/22/2009 JPDO

Requirements Summary• Access

– Aggregate overlapping requests

– High BW and low BW access methods support

• Agility– Ability to add new systems,

services, products, data fields, users, etc. w/o system interruptions

– Support legacy and new systems together

– Scalable over time• Archival / logging

– Past transaction retrieval (e.g., for accident investigations)

Availability Fault tolerance Load balancing Essential FAA service support -

.999 Critical FAA service support

- .99999 SAS (essential):

MTTR - 0.5 hr, MTBF – 5,000 hrs Outage max – 10 mins

SAS (critical): MTTR - 0.5 hr MTBF – 50,000 hrs Outage max – 6 secs

8

Requirements Summary (cont)• Compatibility

– With CWSUs, AWC, WFOs, Tower systems, TRACON systems, ARTCC, ATCSCC

– With FAA architecture• Contents

– Support Wx Products required for aviation purposes, for example:• NOAA provided, FAA-provided, 3rd party

provided• North American and global• Forecasts model data (probabilistic)• Sensor products

– Radar, lightning, satellite, aircraft sensors, airport, ground, ocean, air, METARs

• Observations – PIREPS• Advisories, watches, warnings• Climatological data

– Formats• Grid based (machine readable) where

possible (e.g., NETCDF4)• Encoded versions of legacy text products

(via WXXM or JMBL)• Otherwise, text and graphic? Binary?

Data Consistency Deconflicted SAS (temporal and

spatial) Data Formats / Protocols

Allowing ease of exchange Standards based (e.g. HTTP, XML,

SOAP, OGC) Discoverability

(metadata/registry/ repository) Products/data Formats Access methods Associated services

9

Requirements Summary (cont) Distributed (not centralized) Ease of integration with aviation

decision support tools Governance

Access control Standards Common ontologies Business rules SAS decisions

Intelligent processing Data interpolation

Netcentric SOA based System of systems Legacy system support (along

with new systems) QOS / Performance

Varies by user / use case / function Request management

Request / reply exchange mechanisms

Data for time period of choice Data for geographical construct of

choice Product / data field of choice Priority Desired format / translations Compression

10

Requirements Summary (cont)• Security

– Data security– Physical / network security– Field by field, product by

product access control• Subscription management support• SAS management and governance• System management

– Failure detection / switchover– Load balancing– Health monitoring / analysis /

trending / logging• Users

– Aviation (primarily), governmental, commercial, military, international

– Non-aviation, research, NOAA-internal

• Verification / quality control

11

AssumptionsThe following key assumptions are driving

NOAA’s NextGen IT Architecture:Use of a System of Systems approachCompatibility with key NOAA and NWS

Enterprise Architecture guidanceCompliance with NextGen requirements (with

flexibility to evolve as requirements evolve)Supports IT ConOps Use CasesCompatible with evolving FAA ArchitectureSupportive of NextGen Enterprise Architecture

definition of Business Services / Operational Activities

13

Architecture Efforts to Date• Compiling requirements / use cases• Identifying required Wx Cube data (and candidate source / destination systems), flows, and formats

• Reconciling IOC Product list , FAA Product Flows spreadsheet, SA Plan, others• Working closely with FAA Architecture team to ensure architecture compatibility

• FAA architecture approach – Hub and spoke / Store and forward– Federated registry / repository (for metadata)– FAA Origin Servers (ingest/store/serve FAA data)– FAA Distribution Servers (cache/handle requests of FAA users)– FAA External Distribution Servers (handle requests to and from systems externally to FAA)– Based on WCS/WFS Reference Implementation (RI) being developed by MITLL, NCAR, GSD which will be

available for NOAA re-use• Reconcile JMBL vs. WXXM

• Working with GSD to evaluate use of JMBL – several JMBL RIs may be available for NOAA use• Standards decisions

– Non-gridded– JMBL vs. WXXM (or both) for NOAA (FAA is standardizing on WXXM)

– Gridded data formats – NetCDF4/HDF5, GRIB2? Others?

– Web Services / message exchange standards (e.g, WCS, WFS, JMBL, SOAP, XML, WMS? etc)– Metadata standards

• Beginning to work with NOAA/NWS weather system owners to ensure compatibility of IT architecture / migration approach

• NWS IT System Architecture document and IT System Design document to be developed and released by March 2010 (with interim drafts prior)

14

Sample Wx Products List for NextGen Inclusion NEXRAD Level III Lightning data MADIS (TBD subsets) GOES data METSAT data TAFs METARs/SPECIs SIGMETS / G-SIGMETS AIRMETS / G-AIRMETS Surface fronts Meso-scale boundaries 3-D reflectivity products CCFPs CWAs

• MISs• LAMP output• Tropical cyclone

bulletins• FAs• Tornado watch /

warnings• Severe thunderstorm

watch / warning• Convective outlook• Non-convective

watches, warnings, advisories

• Freezing level analysis• Surface analysis• High level SIG Wx• Verification statistics

• Mid level SIG Wx• Low level SIG Wx• Winds aloft• RUC outputs• WRF-RR outputs• HRRR outputs• NCWF• NCWD• GTG• FIP• CIP• Global wind grids• ASOS OMOs• Others?

15

Resultant Potential Candidate Systems for Cube Inclusion ADDS AWIPS CAWS GIS system IOOS IRIS Lightning Data Services MADIS MDCRS NCDC NCEP/NOMADS

GFS (Global Forecast System) HRRR LAMP NAM (North American

Mesoscale) RUC (Rapid Update Cycle) WRF-RR

• NDFD• NDGD

• NESDIS• GAS (GOES R)• NDE (NPOESS)• Non-NOAA systems

• NEVS (RTVS, Stats on Demand)• NEXRAD (Radar Data Server)• NSSL Mosaics• RIDGE radar servers• TOC (NWSTG/NOAANET)• Web Farms• Other Sources for?

• Canadian Radar• Puerto Rico model data• STMAS data• UKMET model data 16

Layered Architecture Approach

18

Internal FAA Architecture Concept

End Users

ConsumerCube

Service Adaptor(CCSA)

Origin Servers(with

System Ingest Adaptor

and Provider Data)

DistributionServersIP

Network

ProviderSystem

4-D Wx Data Cube

Registry/Repository

ConsumerSystems Net-

EnabledServices

19

Internal FAA Architecture

20

Weather Data Clients

FTI WAN / SWIM

Weather DataOrigin Server

Router


Router

Router


DistributionServer

Registry/Repository

ED-8 Gateway

ITWS ADAS


Router

NWP

DistributionServer

DistributionServer

Consumer Cube Service Adaptors

Consumer Systems




Distribution Servers for

External Access

Consumer Systems

Consumer Systems

DMZ

Definitions Cube Input Edge Server (CIES)

Allows remote access to the weather data (or subsets thereof) via WCS/WFS/other web services. Provides for the ingest of weather data required by the Cube (obtained either directly from the native

source or via a Service Adaptor – which is mostly likely case for most data sources initially) Performs the necessary processing and local storage

Cube Output Edge Server (COES) Provides for the request and retrieval of Cube data from remote WCS/WFS/other web services Performs the necessary processing and local storage Allows access to the data by the requesting local destination system.

Service Adaptor (SA) Source Service Adaptor (SSA) - Performs processing to:

Transform native or legacy source wx data that is required for publishing to the Wx Cube into a format appropriate for ease of access via Cube Input Edge Servers (e.g., transformed into one of several supported standards)

To make source wx data available to Cube Input Edge Servers via a convenient and reliable network accessible means (e.g., Web Services-based communication) where such a means may not currently exist.

Destination Service Adaptor (DSA) - Performs processing to: Transform wx data from a format appropriate for ease of access by Cube Output Edge Servers into a

native or legacy format compatible with the destination system A Service Adaptor may be optional, where all such SA processing may instead be performed directly by a

Cube Input Edge Server or Cube Output Edge Server or internal to the source or destination system themselves (for future systems).

Optional Accessible Storage Optional storage to temporally hold wx data before being ingested into the cube or before being processed

for use by the requesting destination system

24

25


FTI WAN / SWIM


Router


Router

Router


DistributionServer

Registry/Repository

Registry/Repository

ED-8 Gateway

Federated

AWIPS

MADIS

NOMADS

NDFD

NEVS

Lightning Data

Sat Data

Cube Input Edge Server







Wx CubeData

Sources

AWIPSMADIS

Radar Data

Wx CubeData Destinations

Cube Output Edge Server



NWS IP NETWORKS (NOAANET)

NOAA/NWS FAA

CROSS ORGANIZATIONAL NOTIONAL ARCHITECTURE

ITWS ADAS

NWSTG


Router

NWP

DistributionServer

DistributionServer


Consumer Systems




Distribution Servers for

External Access

Consumer Systems

Consumer Systems

DMZ

Example Architecture Use Cases Assumptions

Reg/rep has already been queried in all cases Does not reflect any security aspects

Use Case examples: Requestor-FAA, Data Source – NOAA, Data Type – NonGridded (WFS) Requestor-FAA, Data Source – NOAA, Data Type – Gridded (WCS) Requestor-NOAA, Data Source – FAA, Data Type – NonGridded (WFS) Requestor-NOAA, Data Source – FAA, Data Type – Gridded (WCS) Requestor-NOAA, Data Source – NOAA, Data Type – JMBL Requestor-FAA, Data Source – NOAA, Data Type – NonGridded (JMBL) Requestor-FAA, Data Source – NOAA, Data Type – Gridded (JMBL)

26

33

Optional CIES-Accessible

Storage

Source Service Adaptor (SSA)

Wx Data Source System


WX CUBE

NOAA CUBE SOURCE ALTERNATIVE APPROACHES

Intermediate Stand-alone SA

with CIES

Fully SOA Solution

(embedded CIES)

SOA via External

CIESWx Data Source System

Optional CIES-Accessible

Storage

Source Service Adaptor (SSA)


Intermediate Bolt-on SA with CIES

Net enabled

IF

System Ingest

Adaptor

CIES processing /

storage

Net enabled

IF

System Ingest

Adaptor

CIES processing /

storage

Net enabled

IF

System Ingest

Adaptor

CIES processing /

storage

Net enabled

IF

System Ingest

Adaptor

CIES processing /

storage

CIES

CIES

CIES

CIES


Reg/rep has already been updated in all cases Does not reflect any security aspects

Use Case examples: Loading data into Cube: Source System to CIES via SSA Loading data into Cube: Source System to CIES using Optional Storage Loading data into Cube: Source System to CIES Directly

34

38

NOAA CUBE DESTINATION ALTERNATIVE APPROACHES

Fully SOA Solution

(embedded COES)

Optional COES-Accessible

Storage

Destination Service Adaptor (DSA)

Wx Data Destination SystemNet

enabled IF

System Export

Adaptor

COES processing /

storage

Wx Data Destination

System

WX CUBE

SOA via External COES

Wx Data Destination System

Optional COES-Accessible

StorageWx Data Destination System

Intermediate Bolt-on SA with COES

Destination Service Adaptor (DSA)

Net enabled

IF

System Export

Adaptor

COES processing /

storage

Net enabled

IF

System Export

Adaptor

COES processing /

storage

Net enabled

IF

System Export

Adaptor

COES processing /

storage

Intermediate Stand-alone SA

with COES

Data Request

Data Request

Data Request

COES

COES

COES

COES


Reg/rep has already been queried in all cases Does not reflect any security aspects

Use Case examples: NOAA Cube request via DSA (Intermediate Stand-alone or Bolt-on SA) NOAA Cube request with optional storage (Intermediate Stand-alone or

Bolt-on SA) NOAA Cube request from Destination System (SOA via External COES) NOAA Cube request directly to Data Source (Embedded COES)

39

41

Wx Cube Data

SourceCIES

Wx Cube Data

Source

Wx Cube Data

Source

Wx Cube Data

DestinationCOES

Wx Cube Data

Destination

Wx Cube Data

Destination

Centralized Options

42

Redundancy Options

CIES

Wx Cube Data

Source

CIES

CIES

COES

Wx Cube Data

Destination

COES

COES

Architecture Next Steps• Identifying definitive source for each Cube WX product• Defining Cube “format” for each Wx Product• Resolving JMBL/WXXM standards decision (or decision to support

both) and finalizing all exchange formats / protocols / standards• Involving stewards of each Cube candidate system in determining

details of “connecting” to the Cube• Refining requirements / use cases (performance, security,

verification, varied QOS offerings, external interfaces/system compatibilities, weather products needed)

• Determining key telecommunications infrastructure requirements, interfaces, and implementation details

• Finalizing high level NOAA Cube IT architecture (while ensuring compatibility with FAA architectural approach)

• Developing design for Cube edge servers

• And a million other things to do.

45