status report on noaa’s npoess data exploitation (nde) system

Status Report on NOAA’s NPOESS Data Exploitation (NDE)

System

Presented by Jim Yoe1 Contributions from Geof Goodrum1,

Jim Silva1, and Tom Schott1

1. NOAA/NESDIS Office of Systems Development

NDE Update COPC 10-21-09 WWB2

Agenda

• Background• Status• Hot Topics and Next Steps• Summary


Background

3


Mission & ObjectivesNDE MISSION:

Assist NOAA and other civilian end users to realize the potential of NPOESS observations

Program Objectives:• Coordinate NPOESS-related activities across NOAA

• Assist with planning for implementation of NPOESS data by user systems

• Ensure end-user preparedness for NPOESS data

• Make Operational use of NPP data


Mission & Objectives

Operational System Objectives:

• Disseminate NPOESS Data Records to end users• Generate and disseminate tailored NPOESS Data Records • Generate and disseminate NOAA-unique products (augmented

environmental products constructed from NPOESS Data Records) • Deliver NOAA-unique products, product processing elements, and

associated metadata to CLASS for long-term archiving • Provide services to customers, including NDE product training, product

enhancement across NOAA• Provide software for NPOESS Data Record format translation and other

data manipulations


Status

6


System Development• Completed Software for NPP Ingest Build• Initiated Product Generation Build for NPP• Tested IDPS-NDE Interface with live data• Integrated 3 NUP Algorithm Packages in NDE’s Science

Algorithm Development and Integration Environment• Established NDE Integration Lab (NIL) in NSOF• Tailoring Tool Evaluation/Selection completed• xDR to netCDF4 Translator Prototype delivered• Hardware and COTS SW acquired/installed

─ Dell Blade servers and Dell workstations ─ IBM Blade Center─ StorNext SAN Filesystem


2nd Qtr 3rd Qtr 4th Qtr 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 1st Qtr 2nd Qtr 3rd Qtr 4th QtrCY08 CY09 CY10 CY11 CY12

1st Qtr

NPP Launch“Work To”

16 Jan 2011

NDE SW Build Schedule

NDE’s initial build cycle will support NPP launch

Ingest

Distribution

Customer Services

Monitoring & Reporting

Product Generation

NPP LaunchReadiness Review Oct

2010

Initial Distribution

Capability Aug 2010


Product Development

• FY09 NOAA Unique Product Developments ─ CrIS/ATMS (aka NUCAPS)

Multiple Delivered Algorithm Packages (DAPs) to NDE (Apr, Jul, Aug 2009) Simulated CrIS data delivered to netCDF4 tailoring tool team

─ Sea Surface Temperature Requirements gathered Preliminary Design Review (PDR) slated for December 2009

─ Microwave integrated Retrieval System (MIRS) Successful Preliminary Design Review (PDR) for NPP MIRS Sep 24 DAP successfully integrated into NDE SADIE NESDIS/STAR daily processing ATMS proxy data

─ netCDF4 reformatting Toolkit Draft BUFR tables (ATMS, CrIS, VIIRS) delivered to NCEP for review Successful PDR in May 2009 DAP delivered to NDE in August 2009


Product Development• FY10 NUP Priorities

─ VIIRS Polar Winds Benefit: Numerical weather prediction (NWP)

○ Forecasts can be extended 2-6 hrs, depending on the location, when the MODIS winds are assimilated

○ Hurricane track forecasts are also improved Joint Center for Satellite Data Assimilation (JCSDA) International NWP Centers Satellite product: Atmospheric winds for Arctic and Antarctic regions

─ Green Vegetative Fraction


Systems Integration• Internal Coordination

─ Completed primary ICDs with CLASS and ESPC ─ Established Transition Team to plan for NPP Production─ Communicating w/ NOAA Programs

CoastWatch, AWIPS Tech Infusion, NextGen, HMT, etc.─ Participating in NPOESS TIMs, User Conclaves, Customer Forums─ COPC, CSAB, and JAG/ODAA─ NDE Lessons Learned shared with AFWA for WDA

Requirements, Design, COTs analysis/selections Anticipate sharing system code for evaluation, potential re-use

• Public Outreach─ AMS, AGU, COMET WS, etc.

• Initial Stakeholder Integration Team (SIT) in FY10


NWP SIT Project

• Objective─ Operational NWP user of CrIS

Cloud-Cleared Radiances w/in 1 year of NPP Data Availability

─ Continuity/Expansion of Polar contribution to NWP capability

• Participants─ NDE, JCSDA, NCEP/EMC, STAR,

OSDPD, and IPO• Leverages

─ NDE Phase 1 Products─ NDE Re-formatting Project─ JCSDA ATMS assimilation─ IPO support for post-launch

assimilation of CrIS radiances Sunset this SIT after 2 years

• Approach─ Use cloud clearing to increase

fraction – and impact – of IR radiances used in global model

─ Use AIRS data as CrIS proxy

• Milestones for FY10─ Q1 Experiment Design─ Q1 Experiment Set-up─ Q2 Single season impact

Mid-lat diagnostics─ Q3 Two additional seasons

Full diagnostics incl. tropics─ Q4 Extend to IASI data─ Q4 Prepare NCEP pre-op test


Hot Topics and Next Steps


Hot Topics and Next Steps• NDE Development Contract Transition in FY10

─ Documenting details of system design and configuration─ Documenting skills required for key positions

• NPP Ground System Interface Testing (GSIT)─ Begins November 17 2009

Test Planning Summit currently underway in Suitland, MD─ NDE will only be able to test ingest, not processing─ NDE has only 1 FTE for testing

IPO and NESDIS (ESPC and SSD) providing additional support

• NOAA Mandate to use NPP data as if operational─ NDE to identify/secure resources to harden NDE IT environments─ NDE to Identify/secure resources to accelerate product

development and integration


Hot Topics and Next Steps (2)• Distribution Network Solution

─ Current bandwith out of NSOF too low for complete NPP data load Out of NDE Project Scope

─ NDE advocating for solutions Long term (2013) – NOAA enterprise network capability – C1 Near term – Interim solution to deliver NPP data

○ Upgrade existing link(s)?

• Quality of EDRs and impact on products/applications─ Some NPOESS EDR algorithms outdated or unproven─ NDE Relying on IPO and NESDIS/STAR cal/val to assess impact

and rectify

• Backup Data Access and Processing for C1─ NDE wants to begin serious discussions with AFWA, CSAB


Summary

16


• NDE on schedule and within budget• Good progress in all phases

─ System Development, Product Development, and Integration• Significant challenges remain

─ Potential disruption due to contract transition─ Additional stresses due to “operational production” of NPP data─ Adequacy of network capability for data distribution

• Good cooperation in the context of COPC─ DoD centrals development may be able to re-use NDE elements─ DoD centrals welcome to subscribe to NUPs─ Need to move ahead with planning for Back Up capability for C1

Summary


SoS Work Shop October 6-7, 2009 NDE18

ContactFor more information:

Web http://projects.osd.noaa.gov/nde/

Phone 1-301-817-4418

E-mail [email protected]

http://projects.osd.noaa.gov/nde/



mailto:[email protected]


Back Up Slides


SoS Work Shop October 6-7, 2009 NDE20

NPOESS OverviewNational Polar-orbiting Operational Environmental Satellite System

Space Segment

NPOESS delivers a common set of global products to four US processing centers

State 1: NPP

State 2: NPP & NPOESS C1

State 3: NPOESS C1 &C2

Command, Control, Communications Segment (C3S)

RF

State 1: Single Receptor (Svalbard)

State 2: Multiple Receptors

State 3: 15 Receptors

Interface Data Processing Segment

(IDPS)

State 2,3: 4 sites

APs

APs

APs

State 1: 2 sites

APs

Processing Centers

HDF5

NESDIS

AFWA

NAVOCEANO

FNMOC

HDF5

HDF5

HDF5


NDE Architecture

NDE chose a Service Oriented Architecture to meet data processing requirements

StorageArea Network

IngestCustomer Services

Product Generation Distribution

Products pushed from

NPOESS IDPS

Product User

File Detection

Production Rule Trigger

Registration & Order

Management

Order Profiles

User System


NPP Phase 1─ Legacy mission continuity replacement products comprised of

currently funded NUPs and xDRs

CrIS Thinned Radiances Blended SST Sea Surface Temperature (SST)

CrIS Cloud Cleared Radiances SST Anomalies Nadir Profile Ozone

Total Precipitable Water (ATMS) SST Degree Heating Weeks Ozone Total Column

Snow Cover (ATMS) SST Hot Spots Snow Cover and Depth

Precipitation Rate (ATMS) Coral Reef Bleaching Indices/Alerts Imagery

Land Surface Emissivity (ATMS) Total Ozone (CrIS) Ocean Color/Chlorophyll

Cloud Liquid Water (ATMS) Trace Gases (Carbon) Vegetation Index

Sea Ice Concentration (ATMS) SST (AVHRR-like) Active Fires

Snow Water Equivalent (ATMS) Aerosol (AVHRR-like) Atmospheric Temperature Profile

Ice Water Path (ATMS) ATMS Radiances Atmospheric Moisture Profile

Land Surface Temperature (ATMS) CrIS Radiances Aerosol Optical Thickness

Temperature Profiles (ATMS) VIIRS Radiances Land Surface Type

Moisture Profiles (ATMS) OMPS Radiances Surface Albedo

Rain Water Path (ATMS) Cloud Mask Cloud Cover/Layers

Y NOAA Unique Products (NUPs) B NPOESS Delivered xDRs (SDR, TDR, EDR, ARP, IP)


NPP Phase 2─ Additional legacy products and enhanced products comprised of

NUPs and xDRs not linked to mission continuity

Polar Winds (VIIRS) Fire & Smoke Analysis Cloud Base Height

Green Vegetation Fraction Near Coast Ocean Color Cloud Effective Particle Size

Limb Profile Ozone Harmful Algal Blooms (VIIRS) Cloud Optical Thickness

Blended Snow Cover Clear Sky Radiances (VIIRS) Cloud Top Height (VIIRS)

Blended Total Precipitable Water Integrated xDRs at CrIS Resolution Ice Surface Temperature

Tropical Rainfall Potential Cloud Liquid Water Path (VIIRS) Sea Ice Characterization (VIIRS)

Blended Ozone Cloud Ice Water Path (VIIRS) Atmospheric Pressure Profile

Vegetation Health Cloud Top Temperature (VIIRS) Quarterly Surface Type Gridded

Vegetation Moisture Suspended Matter

Drought Indices Aerosol Particle Size

Vegetation Thermal Conditions Cloud Top Temperature

Leaf Area Index Cloud Top Pressure

Fire Potential/Risk Land Surface Temperature (VIIRS)

Y NOAA Unique Products (NUPs) B NPOESS Delivered xDRs (SDR, TDR, EDR, ARP, IP)


NPP Not In Phase 1 or 2

Net Solar Radiation at TOA (CERES) Rainfall Prediction (ATMS)

Outgoing Longwave Radiation at TOA (CERES) Tropical Cyclone Intensity (ATMS)

Downward Longwave Radiation at TOA (CERES) Volcanic Ash (VIIRS)

Downward Shortwave Radiation at TOA (CERES) Hazard Support (Tropical) (VIIRS)

Cloud Products (CERES) Fire Burn Scars (VIIRS)

Outgoing Longwave Radiation TOA (VIIRS) Cloud Types (AVHRR-like)

Outgoing Longwave Radiation (CrIS) Inversion Strength and Height (CrIS)

Downward Shortwave Radiation TOA (VIIRS) CO2 Slicing Derived Cloud Top Pressure (CrIS)

Ocean Optimized Cloud Mask

Chesapeake Bay Ocean Color

Emiliania Huxleyi Blooms

Y NOAA Unique Products (NUPs)

status report on noaa’s npoess data exploitation (nde) system

Documents

implementation of npoess

nde product training

stepssummarynde update

tailored npoess data

simulated cris data

npptested idpsnde interface

noaaunique products

product enhancement