overview of current s-npp noaa unique products …...product and ancillary data rdrs, sdrs, tdrs,...

Overview of Current S-NPP NOAA Unique Products and an Introduction to STAR's Research to Operations

Transition Process for JPSS-1 ESPC Products

Co-authors Dr. Priyanka Roy1, Walter W. Wolf 2, Thomas King1,

Shanna Sampson1

1IMSG, 2NOAA/NESDIS/STAR

1 97th AMS Annual Meeting 2017 January 24, 2017

13th Annual Symposium on New Generation Operational Environmental Satellite Systems (13NGOESS) and the Seventh Conference on Transition of Research to

Operations Conference (7R2O)

Outline

• Introduction of the Joint Polar Satellite System (JPSS)

• Overview of NOAA/National Environmental Satellite Data and Information Service (NESDIS)’s NOAA-Unique Products (NUPs) generation process.

• Description of the NESDIS/Center for Satellite Application and Research (STAR)’s NUP transition to operations process.

• Role of the Algorithm Scientific Software Integration and

Transition Team (ASSISTT)

97th AMS Annual Meeting 2017 2 January 24, 2017

JPSS Mission

• The Joint Polar Satellite System is the new generation of polar orbiting satellite system, of which Suomi-National Polar-orbiting Partnership launched in 2011 was the first.

• The satellites and the planned instruments within JPSS are:

January 24, 2017 97th AMS Annual Meeting 2017 3

Satellite Launch Year Instruments

S-NPP 2011

Visible Infrared Imaging Radiometer Suite (VIIRS), Advanced Technology Microwave Sounder (ATMS), Ozone Mapping and Profiler Suite Limb and Nadir (OMPS-L and OMPS-N) , Cross-track Infrared Sounder (CrIS) and Clouds and the Earths Radiant Energy System –Flight Model5 (CERES-FM5)

JPSS-1 4th Qtr FY2017 VIIRS, OMPS-N, ATMS, CrIS and CERES-FM6

JPSS-2 1st Qtr FY2022 VIIRS, OMPS-N, ATMS, CrIS and Radiation Budget Instrument (RBI)

JPSS-3 FY2026 VIIRS, OMPS-N, ATMS, CrIS and RBI

JPSS-4 FY2031 VIIRS, OMPS-N, ATMS, CrIS and RBI

Source : http://www.jpss.noaa.gov/launch_schedule.html

NOAA/NESDIS’s S-NPP & JPSS-1 Products

NESDIS generates higher level products from VIIRS, ATMS, OMPS and CrIS by processing the Raw Data Records (RDRs) :

– The Interface Data Processing Segment (IDPS) • Sensor Data Records (SDRs) • Temperature Data Records (TDRs) • Environmental Data Records (EDRs)

– The NPP Data Exploitation (NDE)

• NOAA Unique Products (NUPs) – All products created within NDE are considered NUPS

• Tailored Products Both IDPS and NDE have been providing critical S-NPP data operationally since 2011.

The entire product generation and distribution system is currently being updated to provide an enterprise solution for product generation from multiple satellites (Transition to Operations (TTO) date – March 2017).


Overview of ESPDS Operations in JPSS Common Ground System (CGS) IDPS Block 2.0 era


JPSS CGS IDPS

S-NPP, JPSS-1

Svalbard/McMurdo

Environmental Satellite Processing and Distribution System (ESPDS)

NDE – Product Generation Enterprise Shared Storage

Product Distribution and Access

External and

Internal Users

National Centers for

Environmental Information

(NCEI) Archive

xDRs, NUPs, Tailored Products

NUPs

xDRs, NUPs, Tailored Products

Ancillary Data file Availability Notification

Product and Ancillary Data files

Product and Ancillary Data files

RDRs, SDRs, TDRs, EDRs

Currently, with JPSS CGS IDPS Block 1.2, NDE distributes only the NUPs and Tailored Products to users and NCEI for archive using its Product Distribution Services.

TTO is scheduled for March 2017

S-NPP Products after Block 2.0 TTO

Products that are already operationally available from NDE 1.0 – Atmospheric Vertical Moisture Profile (CrIS/ATMS) (NUCAPS) – Atmospheric Vertical Temperature Profile (CrIS/ATMS) (NUCAPS) – Infrared Ozone Profile (CrIS) (NUCAPS) – Outgoing Longwave Radiation (CrIS) (NUCAPS) – Green Vegetation Fraction (VIIRS) Global (NetCDF) – Green Vegetation Fraction (VIIRS) Regional (NetCDF) – Green Vegetation Fraction (VIIRS) Global (GRIB2) – Green Vegetation Fraction (VIIRS) Regional (GRIB2) – Vegetation Health Index Suite (VIIRS) – VIIRS Polar Winds (VIIRS) – Cloud Liquid Water (ATMS) (MiRS) – Imagery (ATMS) (MiRS) – Land Surface Emissivity (ATMS) (MiRS) – Land Surface Temperature (ATMS) (MiRS) – Moisture Profile (ATMS) (MiRS) – Rainfall Rate (ATMS) (MiRS) – Sea Ice Concentration (ATMS) (MiRS) – Snow Cover Depth (ATMS) (MiRS) – Snow Water Equivalent (ATMS) (MiRS) – Temperature Profile (ATMS) (MiRS) – Total Precipitable Water (ATMS) (MiRS) – Sea Surface Temperature (VIIRS) ACSPO – Active Fires (VIIRS)


* More information about these products is provided in the poster titled “JPSS Environmental Satellite Processing Center Data Products” - Bonnie Reed and T. Schott, A. Layns, and S. L. Bunin (1336-4E)


Additional products that will be available with NDE 2.0 – Cloud Mask (VIIRS) – Cloud Phase (VIIRS) – Cloud Cover Layers (VIIRS) – Cloud Height (Top and Base) (VIIRS) – Cloud Top Pressure (VIIRS) – Cloud Top Temperature (VIIRS) – Cloud Type (VIIRS) – Cloud Optical Depth (VIIRS) – Cloud Particle Size Distribution (VIIRS) – Cloud Ice Water Path – Cloud Liquid Water (VIIRS) – Volcanic Ash Detection and Height (VIIRS) – Aerosol Detection (VIIRS) – Aerosol Optical Depth (VIIRS) – Aerosol Particle Size (VIIRS) – Snow Cover (VIIRS) – Ice Concentration (VIIRS) – Ice Age/Thickness (VIIRS) – Ice Surface Temperature (VIIRS)




Deferred Products – will become available in 2017 – Albedo (Surface) EDR (VIIRS) – Land Surface Temperature EDR (VIIRS) – Surface Reflectance EDR (VIIRS) – Vegetation Indices EDR (VIIRS) – Carbon Dioxide (CO2) (CrIS) (NUCAPS) – Carbon Monoxide (CO) (CrIS) (NUCAPS) – Methane (CH4) (CrIS) (NUCAPS) – Ocean Color/Chlorophyll (VIIRS) – OMPS Limb Profile EDR – OMPS Limb Profile SDR – Ozone Nadir Profile (OMPS-N) – Ozone Total Column (OMPS-N)



STAR’s Science Algorithm Transition to Operations Process for NUPS


SPSRB Process

• The NUPs follow the Satellite Products and Services Review Board (SPSRB) review process for transitioning products into operations.

• The SPSRB process has been updated following information from STAR’s Enterprise Product Lifecycle (EPL) Process which is a Capability Maturity Model Integration (CMMI) Level -3 Process.

• This SPSRB process is an established and mature process for efficiently and successfully transitioning science algorithms (SAs) to operations.

• The SPSRB Process Improvement Working Group (SPIWG) oversees the process.


* More information about these products is provided in the presentation titled “Transitioning NOAA Satellite Products from Development to Operations” - Stacy L. Bunin and K. Shontz and B. Reed (5.6 -34B)

SPSRB Standards and Guidelines

• Standards & Guidelines – Process Review Standards – NDE Delivered Algorithm Package (DAP) Standards is an ICD between STAR

and NDE. – SPSRB Coding and Scripting Standards – Data Format Standards

• Archive Guidance – Guidance on NCEI Interactions

– NCEI uses multiple storage systems to archive data including the Comprehensive Large Array-Data Stewardship System (CLASS), which is jointly governed by NCEI, OSGS and OSPO.

– NCEI has laid down procedures to provide guidance to STAR and OSPO personnel for archiving satellite data and products adhering to the NOAA Procedure for Scientific Records Appraisal and Archive Approval (http://www.nosc.noaa.gov/docs/products/NOAA_Records_Brochure_4_pages_Dec_9.pdf ).


Project Documentation

• SPSRB Documentations – Generated and maintained by Development Team – Algorithm Theoretical Basis (ATBD) – External Users Manual (EUM) – Internal Users Manual (IUM) – System Maintenance Manual (SMM)

• Review Process Documentations – Generated and maintained by STAR

ASSISTT – Requirements Allocation Document (RAD) – Review Item Disposition Spreadsheet (RID)


Key Stakeholders for the NUPs

– NOAA JPSS Office (NJO) • Provides funding • Defines the Level 1 Requirements for the products

– Office of Satellite Ground System (OSGS) • Enables development of satellite products and their transition to operations

– Center for Satellite Applications and Research (STAR) • Develops satellite products ensuring they meet all product and process requirements

– Office of Satellite and Product Operations (OSPO) • Manages and directs the operations of the central ground facilities which ingest, process and

distribute environmental satellite data and derived products

– National Centers for Environmental Information (NCEI) • Archives the satellite data and products

– Users • NOAA users, DoD, International Partners


NUP TTO Process

97th AMS Annual Meeting 2017 14

Developmental Phase

Process Reviews • Requirements Review (STAR) • Preliminary Design Review (STAR) • Critical Design Review (STAR) • Unit Test Readiness Review (STAR) • Software Code Review (OSPO) • Algorithm Readiness Review (STAR)

Tasks • Initial Archive Requirements identified • Quality Monitoring Concept Defined • Long-term Maintenance Concept Defined • Development processing system defined • Initial Information Technology (IT) Security concept defined • Test case processed • Code is prepared for implementation • Final Archive requirements identified • Operational and backup processing defined

Pre-Operational Phase

Process Reviews • Operational Readiness Review (OSPO) • SPSRB Briefing (OSPO and STAR)

January 24, 2017

Tasks • Operational and backup processing capabilities in place • Final IT Security Concept Defined • Pre-operational product output evaluated & tested • Code transitions to operations; all documentation is complete • Operational and backup capabilities reach ops status

ASSISTT

• ASSISTT is the bridge between science at NOAA/NESDIS/STAR and operations. Enforce a standard

software quality. Transparently

communicate between science and operations.

Science • Algorithm Design

• Prototype • Improve • Validate

ASSISTT • Research to Ops

• Standardize • Optimize • Quality Assurance

Operations • Product Execution

• Generate • Monitor • Distribute

ALGORITHM SCIENTIFIC SOFTWARE INTEGRATION AND SYSTEM TRANSITION TEAM

15 January 24, 2017 97th AMS Annual Meeting 2017

The Problem Many scientists prefer interpreted

languages (IDL and Matlab) or even older languages (Fortran 77). Fortran 90/95 or C/C++ expertise does not always exist. Code doesn’t account for operational

concerns such as run time, memory usage, disk I/O, error checking. Code is often written by a mixture of

programmers with varying styles and abilities.

Paths are hardcoded and algorithms

assume the data they need will be in arranged in a particular data tree. Executable code makes system calls

(assuming a certain OS). Code is often not well documented. Code only works with certain

compilers. Code uses non-standard functions.

Operations is tasked only to receive, generate, and monitor products – not upgrade code to operational standards.

Scientists write code that gets transitioned to operations, but:


Role of STAR ASSISTT

Act as the “Middle Man” • Trace all requirements and document and update all the requirements in the

project RADs • Perform product and process QA. • Lead all reviews • Provide software guidance associated with the algorithm development and

updates • Provide Algorithm Package guidance • Test and verify DAPs before delivery to NDE • Provide reporting to management • Provide coordinated schedules and schedule oversight • Add algorithm software to STAR CM system • Deliver the preliminary and final DAPs to NDE


STAR ASSISTT

STAR Science Team NDE DAP DAP

ASSISTT Staff

• ASSISTT Project Manager • Integration Leads • QA Leads • Algorithm Integrators • Algorithm Change Experts • Scripting and Transition Experts • Regression and System Testers • Configuration Manager • IT Cluster Managers


Summary

• JPSS has established an enterprise approach for satellite product generation.

• The updated ground system is getting ready for transition to operations for generating and distributing products to users.

• STAR ASSISTT is all set to support the overall efforts with its tried and tested pathway for transitioning products swiftly and efficiently into operations.


Backup


JPSS ESPC Products Project Element Observational Parameter Project Element Observational Parameter

NUCAPS CrIS Radiance Thinning Polar Winds VIIRS Polar Winds

NUCAPS Temperature Profiles Ocean Color/Chlorophyll Ocean Color

NUCAPS Moisture Profiles Ocean Color/Chlorophyll Optical Properties - Absorption

NUCAPS Infrared Ozone Profile Ocean Color/Chlorophyll Optical Properties - Backscattering

NUCAPS CO2 Ocean Color/Chlorophyll Optical Properties – Chlorophyll Fluorescence

NUCAPS CO Ocean Color/Chlorophyll Chlorophyll

NUCAPS CH4 ACSPO Sea Surface Temperature

OLR CrIS Outgoing longwave radiation Active Fires Fire Geolocation

MiRS Land Surface Temperature Active Fires Fire Radiative Power

MiRS Cloud Liquid Water Active Fires Fire Mask

MiRS Rainfall Rate Green Vegetation Fraction Green Vegetation Fraction – Global

MiRS Sea Ice Concentration Green Vegetation Fraction Green Vegetation Fraction – Regional

MiRS Snow Cover Vegetation Health Suite Vegetation Condition Index

MiRS Snow Water Equivalent Vegetation Health Suite Temperature Condition Index

MiRS Total Precipitable Water Vegetation Health Suite Vegetation Health Index

MiRS Land Surface Emissivity

MiRS Moisture Profile

MiRS Temperature Profile


JPSS ESPC Products

Project Element Observational Parameter Project Element Observational Parameter

Total Ozone Total Column Ozone MTCP Tropical Cyclone Maximum Wind

Ozone Nadir Profile Ozone Nadir Profile MTCP Minimum Sea Level Pressure

Surface Reflectance Surface Reflectance MTCP Radii of 34 kt winds in NE, SE, NW, and SW

Vegetation Indices Vegetation Indices MTCP Radii of 50 kt winds in NE, SE, NW, and SW

Reformatting Toolkit CrIS Radiances (BUFR) Reformatting Toolkit SST (BUFR)

Reformatting Toolkit ATMS Radiances (BUFR) Reformatting Toolkit Aerosol Optical Depth (BUFR)

Reformatting Toolkit VIIRS Radiances (BUFR) Reformatting Toolkit VIIRS Polar Winds (BUFR)

Reformatting Toolkit Nadir Profile Ozone (BUFR) Reformatting Toolkit GVF (GRIB2)

Reformatting Toolkit Total Column Ozone (BUFR)


overview of current s-npp noaa unique products …...product and ancillary data rdrs, sdrs, tdrs,...

Documents