prepared by: tom king 1 , zhaohui cheng 1 , larisa koval 1 , and yi song 2 , 1 psgs, 2 imsg

1

NetCDF4 Reformatting Toolkit (N4RT): BUFR and GRIB2

Tailoring Critical Design Review

September 14, 2009

Prepared By: Tom King1, Zhaohui Cheng1,Larisa Koval1, and Yi Song2,

1 PSGS, 2 IMSG

2

Review Agenda

Introduction 9:00 am – 9:20 am ChengPDR Report 9:20 am – 9:40 am ChengRequirements 9:40 am – 10:10 am KingSoftware Architecture 10:10 am – 10:45 am KingQuality Assurance 10:45 am – 11:05 am KingRisks and Actions 11:05 am – 11:20 am ChengSummary and Conclusions 11:20 am – 11:30 am King

3

Review Outline

Introduction PDR Report Requirements Software Architecture Quality Assurance Risks and Actions Summary and Conclusions

4

Introduction

Presented by

Zhaohui ChengNOAA/NESDIS/STAR

5

Introduction Project Background

» IJPS» NPP/NPOESS» NDE

Project Objectives

Integrated Product Team

Project Plan

Entry and Exit Criteria

6

Project BackgroundNPP/NPOESS

• NPP and NPOESS, a joint Military/NOAA/NASA effort, is the next series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the NPOESS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (1:30 pm).

• Instrument packages on NPOESS:» CrIS, ATMS, VIIRS, OMPS, SEM, CERES, MIS

• NPP is the first of five missions with launch dates of ≈2011, ≈2013, ≈2016, ≈2018, ≈ 2020, respectively.

7

Project BackgroundNDE

Disseminate NPOESS Data Records to customers.

Generate and disseminate tailored NPOESS Data Records (versions of NPOESS Data Records in previously agreed alternative formats and views).

Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPOESS Data Records).

Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving.

Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA.

Provide software for NPOESS Data Record format translation and other data manipulations.

8

To build a software package that will tailor NPOESS and NDE products from NetCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts.

The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products.» Flexible» Extendable

The software must be able run within the NDE system architecture and operate within the NDE functional guidelines.

Output product formats and content must meet the needs of NOAA customers.

Project Objectives

9

Products to Reformat» ATMS Radiances » CrIS Radiances » Nadir Profile and Total Column Ozone (OMPS)» VIIRS Radiances » Aerosol Optical Thickness » Sea Surface Temperature » Snow Cover (Currently Tabled)» Vegetation Index (Currently Tabled)» Polar Winds (Possibly New)» Green Vegetation Fraction (Possibly New)

Project Objectives Phase 1 Products

10

Integrated Product Team

IPT Lead: Walter Wolf (STAR)

IPT Backup Lead: AK Sharma (OSDPD)

NESDIS team:» STAR: Walter Wolf, Hank Drahos, Jaime Daniels, Yi Song, Thomas King,

Larisa Koval» OSDPD: Dave Benner, AK Sharma, Ricky Irving» OSD: Tom Schott, Jim Yoe» Data Center: Lei Shi (NCDC)

User team» Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber

(NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard (JCSDA), Tony McNally (ECMWF), Fiona Hilton (UK-Met)

» Others: International NWP users, NWP FOs, Climate Users

Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP

11

Project Stakeholders

NOAA National Weather Service» Weather Forecast Offices» National Center for Environmental Prediction

Department of Defense» NRL» FNMOC» AFWA

Global NWP» EUMETSAT» ECMWF» UK Met» Meteo France» CMC

12

Project Plan Year 1 – Design and Development (2008 – 2009)

» Verify Requirements– Work with customers to verify product requirements– Discuss with the current developers of similar translators to

determine what is required in their output files» Design the NetCDF4 reformatting toolkit;» Conduct PDR» Develop BUFR tables and GRIB2 formats with the product teams

for Phase 1 products» Work with NDE to determine the interface between the Level 1B

and the Level 2 NPP products and the reformatter» Conduct CDR

13

NetCDF4 Reformatter: System Design and

Development for Phase 1

14

Project Plan

Year 2 –Transition to Pre-Operations of Phase 1 Products (2009 – 2010)» Set up infrastructure to implement the readers and

writers for the data formats» Implement BUFR tables and GRIB2 formats for the

Phase 1 products on the NDE hardware» Conduct Test Readiness Review for Phase 1 products» Transition and test system within the NDE environment» Conduct Code Review for Phase 1 products

15

Project Plan

Year 3 – Transition to Operations of Phase 1 Products (2010 – 2011)» Evaluate with NDE and OSDPD the implementation of

the Reformatting Toolkit within the NDE data handling system

» Conduct System Readiness Review for Phase 1 products

» Transition pre-operational Phase 1 product reformatting system to operations

16

Phase 1 Transition to Operations

17

Project Plan – Schedule

Schedule (Milestones)» Project begins – 07/01/08» Preliminary Design Review – 04/14/09 (10/21/08)» Critical Design Review – 09/14/09 (03/19/09)» Test Readiness Review – 04/14/10 (02/25/09)» Code Unit Test Review – 05/12/10 (01/29/10)» System Readiness Review – 01/31/11 (04/20/10)–Waive or shift to NDE

18

Entry Criteria PDR Completed and the following items reviewed:

» Requirements» Software Architecture » Quality Assurance» Risks and Actions

Requirements Allocation Document - Updated

PDR Report» The PDR Report (PDRR) is a standard artifact of the STAR EPL

process. The PDR report is produced after the PDR. It contains:– Actions– Comments– Revision of the PDR

» PDR Report for this project contains 13 items for review. These will be discussed in the next section.

19

Exit Criteria Conduct the CDR

The CDR Report (CDRR) is a standard artifact of the STAR EPL process. The CDR report is produced after the CDR. It contains:» Actions» Comments» Revision of the CDR

20

Summary of Review Objectives

Review the Project Schedule and Plans

Review of the PDR Report

Review the Requirements

Review Software Architecture

Review Quality Assurance

Identify Risks and Actions

21

Review Outline


22

PDR Report

Presented by


23

PDR Risk Risk 1: NPOESS product formats and content are still changing,

especially for VIIRS Assessment: Low Impact: May have to revise software several time during development

to adjust to new formats, names, and types. Likelihood: High Mitigation: Work through the NPOESS Data Format Working Group

to obtain information on format and algorithm updates. Monitor the latest copies of the NPOESS Common Data Format Control Books (CDFCB) for any updates. Maintain contact with customers to inform them of any upstream product changes. Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision.

Status: Open

24

PDR Risk Risk 2: The roles and responsibilities regarding who shall

generate the set of required SPSRB documents for NDE has not yet been decided.

Assessment: Low Impact: Difficult to budget time needed for the team to

generate documentation. Likelihood: Moderate Mitigation: This issue, and that of document content, is

being worked by Maurice McHugh, STAR, NDE, OSD, and OSDPD personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions.

Status: Open

25

PDR Risk Risk 3: There are small variations in the types of platforms

and the versions of the compilers Assessment: Low Impact: May obtain different results using different

compilers. Likelihood: Moderate Mitigation: Reformatting Toolkit developers will work with

NDE during system tests in the integration and production phase to ensure that those results match the results from the units. The BCR process (mentioned earlier) should help to resolve these issues early on in development.

Status: Open

26

PDR Risk

Risk 4: Data format translation involves some unit conversion and possible reduction of precision (significant digits)

Assessment: Low

Impact: Any modification of the data from its original form may not be apparent to the user.

Likelihood: Low

Mitigation: Document data manipulations in the NDE Delivered Algorithm Package (in place of ATBD).

Status: Open

27

PDR Risk

Risk 6: Risk on NDVI and snow mask. Test GRIB2 functionality before/for CDR.

Assessment: Moderate Impact: Failure to meet the requirement to have

demonstrated GRIB2 tailoring capability. Likelihood: Low Mitigation: RAD/Requirements meeting before

CDR. Status: Open

28

PDR Risk Risk 8: Requirement to write tailoring into GRIB2 VIIRS snow cover

and vegetation index. Need to insure that view point M. EK is known and understood by EMC director.

Assessment: Moderate Impact: User (EMC) does not receive the product they requested. Likelihood: Moderate Mitigation: Yoe/Schott to contact M. EK and coordinate w/s

Lord/EMC/JCSDA before decisions (i.e. snow cover and vegetation index may decide differently)

Status: Open Comments: Has this been done? If so, it can be closed.

29

PDR Risk Risk 9: NWS and other NWP users will want ATMS/CrIS spectral

response functions. They will also want OMPS key calibration functions. This information appears to be contained In ITAR document that we cannot share with foreign users.

Assessment: Moderate Impact: Users will not be able to use the data. Likelihood: High Mitigation: NDE should work with IPO to determine if we can receive

and release that ATMS, CrIS and OMPS SPF and KCF data to foreign NWP users. It is our understanding that the information is in ITAR documentation

Status: Open Comments: We recommend closing this risk since it is not a risk to

the Tailoring project. It is a user need that would be best addressed by IPO.

30

PDR Risk Risk 11: A lot of effort would be imposed on projects/NDE to convert

to NetCDF4 (depend heavily on NDE schedule for the conversion to NetCDF4).

Assessment: Low Impact: Increased schedule pressure on NDE Product Teams. Likelihood: Low Mitigation: Make converter tool flexible to accept other formats not

just NetCDF4. Those formats should include:1) project formats (MIRS, NUCAPS, Ozone etc.) 2) HDF5 format from IDPS. This is not a risk for the Tailoring project. This is an NDE or Product Team risk.

Status: Open Comments: We recommend closing this since it is not a risk to the

Tailoring project. It’s a risk to NDE and the Product Teams.

31

PDR Risk Risk 12: Timelines gets tight If we have to convert HDF5,

project format to NetCDF4 and then to other format. Assessment: Moderate Impact: Users may receive their product later than

desired. Likelihood: NA Mitigation: Asses timeliness issue (due to added

conversion step). Status: Open Comments: We recommend closing this since it is not a

risk to the Tailoring project. It’s a risk to NDE and the Product Teams.

32

PDR Comment Review Item 5: Requirement section: traceability to L1

requirements is a strength. This section mingles project requirements, system requirements and assumptions.

Mitigation: In the RAD and PDR, clearly identify assumptions up-front, including expectations of the NDE system. Remove "NDE shall…" requirements. NDE team will validate assumption and assume new requirements if necessary.

Comments: The PDR and RAD have been revised to identify only the Tailoring project’s requirements. The language in the requirements analysis has been modified to express the project’s understanding of NDE’s roles.

33

PDR Comment

Review Item 7: Set up a meeting with Tom Schott on new aerosol product requirements/requests.

Comments: Has this been done?

34

PDRR – Item 10

Review Item 10: Minor terminology and details. 1) IPs: Not all IPs are created equal. DIPs: Delivered IPs (OMPS NP O3 Profile). RIPS: Retained IPs 2) O3 POP is now Atmospheric Chemistry POP. 3) OMPS Total Ozone Algorithm is multiple triplet Not v8 primary. Output fields are equivalent to v8 on GOME-2.

Mitigation: 1) Not existence of DIPs. Current OMPS DIP is v6. 2) Change in POP list. 3) Get sample EDR compare fields to GOME-2 v8 PMF

Comments: The PDR has been updated to contain the enhanced IP definitions and ozone algorithm version.

35

PDR Comment

Review Item 13: Good idea to have placeholder for additional quality flags (for land mask etc.). Filling them depends on timelines. Why not the same thing is done to ATMS BUFR table.

Mitigation: Plan for placeholders of additional quality flags for ATMS radiance BUFR table. The need maybe there but not expressed.

Comments: There are already spare bit fields in the ATMS BUFR quality flag fields if new ATMS quality flags become available. With respect to the tailoring software, it’s not within the scope of this project to assess data quality such that we generate and add to the BUFR our own new quality information.

36

PDRR Summary

There are 9 risks and 4 comments at this time.

5 risks are rated as low, 4 as moderate.

We recommend closing 3 of the risks.

6 risks will remain open.

An additional risk may be closed depending upon the status of progress (meetings Yoe/Schott and EMC).

37

Review Outline


38

Requirements

Presented by

Thomas KingNOAA/NESDIS/STAR

39

Requirements Overview SPSRB Requirements were presented to the developers in a document

entitled: “Level 1 Requirements for a NetCDF4 Reformatting Tool” (Version 1.5).

Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, EMC (the customer) and the heritage product teams.

Using all of this information a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project.

Text highlighted in yellow indicates requirements that have been updated or refined since the PDR.

40

Phase I User-to-STAR Mapping

Prioritized Product EMC User Contacts

STAR/OSDPD Product and Heritage Contacts

Heritage Product

ATMS Radiances Dennis Keyser, John Derber

Dennis Keyser, John Derber, Sid Boukabara

AMSU, AMSR-E

CrIS Radiances Jim Jung, Dennis Keyser

Jack Woollen, Simon Elliott, Tom King

IASI, AIRS

VIIRS Radiances Dennis Keyser, John Derber

Dennis Keyser, John Derber

AVHRR GAC

Nadir Profile and Total Column Ozone (OMPS)

Dennis Keyser Larry Flynn, Donna McNamara

SBUV, GOME

Aerosol Optical Thickness Jeff McQeen Paul Haggerty MODIS Aerosols

Sea Surface Temperature Bert Katz, William Gemmill

Shasha Ignatov, John Sapper, Robert Grumbine

AVHRR derived SST (ACSPO)

Green Vegetation Fraction Michael Ek Ivan Csiszar, Hanjun Ding AVHRR Veg.

Polar Winds TBD Jeff Key, Jaime Daniels AVHRR and MODIS

41

Functional Requirements:Reformatting Toolkit

Software Requirement: STAR shall deliver to NDE a

reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2).

» Requirement: The toolkit shall be capable of reformatting the NPP tailoring prioritized phase 1 product list.

42


Software» Requirement: The toolkit shall provide its capabilities such that it may be run

automatically within an operational system, especially within the NDE environment. – The Toolkit shall compile and run on the NDE IBM AIX P5 series

hardware.– The Toolkit shall interact with the NDE Data Handling System (DHS).– The Toolkit shall be able to read a Production Control File (PCF).– The Toolkit shall handle and return errors according to NDE/STAR

standard codes.– The Toolkit shall be able to write a (Product Status File) PSF.

» Requirement: The toolkit shall consist of modular components that can be tested independently. – The code shall consist of a single compiled program that parses

arguments and logically assigns tasks to a family of hierarchically structured tailoring subroutines.

– Data shall be stored in allocatable data structures.

43


Software» Requirement: STAR shall include one update to the reformatting

toolkit within its initial project plan.

» Requirement: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products.

» Requirement: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit.

» Requirement: STAR shall update the reformatting toolkit when NCEP updates its BUFR and GRIB2 libraries.– Updates shall be made when there are updates to the versions of

the NetCDF4 library being used by NDE.

44


Software» Requirement: STAR shall coordinate with the NDE Project before

proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development.

» Requirement: The output from the toolkit shall be compared with the input.

» Requirement: The translation toolkit shall convert from the new format back into NetCDF4.

» Requirement: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes. – The Reformatting Toolkit software shall generate run logs and

return NDE/STAR standard (agreed upon) error codes to the DHS.

45

Functional Requirements:

Reformatting Toolkit Software

» Requirement: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination).

» Requirement: The toolkit parameters (e.g., how to use the service) shall be well documented. – Reformatting Toolkit Developers shall provide documentation in

the form of a tailored Delivered Algorithm Package (DAP).

» Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators. – Reformatting Toolkit shall use the standard set of error return

codes developed by NDE for code running within the DHS.

46

Functional Requirements:

Reformatting Toolkit Software

» Requirement: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting. – All messaging shall be directed to a run log file. These messages

shall be documented in the Reformatting Toolkit tailored DAP.

» Requirement: STAR shall coordinate development of the reformatting toolkit with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system.

» Requirement: Toolkit code shall adhere to the STAR coding standards.

» Requirement: Performance shall be measured on a product level.

47

Program Requirements:Reformatting Toolkit

Project

Requirement: STAR shall provide monthly project status reports to OSDPD and OSD.

Requirement: Earned Value Management shall be performed on the project.

Requirement: STAR shall update the project plan on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration.

Requirement: The toolkit shall be implemented and tested six months before the NPP launch to ensure NDE readiness.

48

Product Requirements:CrIS Radiances

Requirement: The Reformatting Toolkit shall tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC.» The Reformatting Toolkit developers shall work with EMC to

create a BUFR table for the NUCAPS thinned radiances based on AIRS and IASI.

» The table shall use delayed replication for storing the radiances.» BUFR messages shall be smaller than 50KB.» The BUFR format shall allow for the storage of negative

radiances.» The file shall contain the following data fields (see table next

slide):

49

Product Requirements:CrIS Radiances

Variable Name Variable Name Variable Name Variable Name

Satellite ID Latitude Height of Land Surface

Start Channel (per band)

ID of Originating Center

Longitude Satellite Height End Channel (per band)

Satellite Instrument Satellite Zenith Angle Land Fraction Calibration Quality Flags

Satellite Classification

Satellite Azimuth Land/Sea Qualifier Field of View Quality Flags

Year Solar Zenith Cloud Cover Geolocation Quality

Month Solar Azimuth Height of Cloud Top NUCAPS Quality

Day Ascending/Descending flag

Radiance Type Flags Channel Number

Hour Scan Line Number Scan-Level Quality Flags

Channel Radiance

Minute Field of Regard Type of Band

Second Field of View Starting Wavenumber (per band)

Location of Platform Orbit Number Ending Wavenumber (per band)

50

Product Requirements:ATMS Radiances

Requirement: The Reformatting Toolkit shall tailor the NPOESS ATMS SDR and TDR data from NetCDF4 into BUFR for EMC. » The ATMS BUFR file shall contain, from all channels, the antenna and

brightness temperatures, associated Quality Flags, and the geolocation data at native resolution (not resampled) data.

» The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is currently provided for AMSU and MHS.

» BUFR messages shall be smaller than 50KB.» The file shall contain the following data fields (see table next slide):

51

Product Requirements:

ATMS RadiancesVariable Name Variable Name Variable Name

Satellite ID FOV Number ATMS Channel Number

ID of Originating Center Granule-Level Quality Flags ATMS Central Frequencies

Satellite Instrument Scan-Level Quality Flags ATMS Channel Bandwidth

Satellite Classification Geolocation Quality Polarization

Year Latitude Antenna Temperatures

Month Longitude Brightness Temperatures

Day Satellite Height Channel-Level Quality Flags

Hour Satellite Zenith Angle

Minute Satellite Azimuth

Second Solar Zenith

Scan Line Solar Azimuth

52

Product Requirements:OMPS Ozone

Requirement: The Reformatting Toolkit shall tailor NPOESS OMPS Ozone products from NetCDF4 into BUFR for EMC.» The product shall contain OMPS Nadir Profile and Total

Column (multiple triplet algorithm, not v8, but output is equivalent to v8 according to Larry Flynn).

» The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV.

» BUFR messages shall be smaller than 50KB.» This file’s content is currently under development.

53

Product Requirements:VIIRS SST

Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS SST products from NetCDF4 into BUFR for EMC.» Product shall contain Skin SST, Bulk SST, Quality Flags, Cloud Mask,

and geolocation data.» Reformatting Toolkit developers shall work with EMC to create a BUFR

table for the VIIRS SST product. » The VIIRS SST BUFR table shall be derived from that currently being

used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans).

» BUFR messages shall be smaller than 50KB.» The file shall contain the following data fields (see table next slide):

54


VIIRS SST

Variable Name Variable Name Variable Name

Satellite ID Latitude Cloud Mask


Longitude Adjacency Cloud Mask

Satellite Instrument Satellite Zenith Angle SST Pixel-Level Quality flag

Year Satellite Azimuth SST (skin)

Month Solar Zenith SST (skin) Quality

Day Solar Azimuth SST (bulk)

Hour Satellite Height SST (bulk) Quality

Minute Geolocation Quality

Second VIIRS Geolocation Quality

55

Product Requirements:VIIRS Radiances

Requirement: The Reformatting Toolkit shall tailor NPOESS VIIRS Radiances from NetCDF4 into BUFR for EMC.» Each BUFR file will contain the VIIRS data for a single band (Imagery

band, Moderate band, or Day/Night band resolution).» Coverage will be global.» Each of the 3 file types will use the same VIIRS BUFR table.» The product shall contain the land and cloud mask if it doesn’t take too

long for the IDPS to generate those EDRs. » Reformatting Toolkit developers shall work with EMC to create a VIIRS

BUFR table derived from that used earlier for the GAC AVHRR.» BUFR messages shall be smaller than 50KB.» The file shall contain the following data fields (see table next slide):

56


VIIRS Radiances


Satellite ID Latitude VIIRS Geolocation Quality


Longitude Radiance Quality

Satellite Instrument Satellite Zenith Angle Cloud Mask

Year Satellite Azimuth Surface Type

Month Solar Zenith Channel Number

Day Solar Azimuth Channel Wavelength

Hour Satellite Height Channel Radiance

Minute Type of Band Channel Reflectance

Second Geolocation Quality

57

Product Requirements:Aerosol Optical Thickness

Requirement: The Reformatting Toolkit shall tailor NPOESS Aerosol Optical Thickness (AOT) from NetCDF4 into BUFR for EMC.» The product shall contain the AOT, wavelength of AOT,

and Aerosol Size.» Reformatting Toolkit developers shall work with EMC to

develop the AOT BUFR table based on what has already been done for MODIS.

» BUFR messages shall be smaller than 50KB.» The file shall contain the following data fields (see table

next slide):

58

Product Requirements:VIIRS Aerosol Optical

Thickness


Satellite ID Latitude Retrieval Quality


Longitude Surface Type

Satellite Instrument Satellite Zenith Angle Aerosol Type (land)

Year Satellite Azimuth AOT Quality Flag

Month Solar Zenith Aerosol Wavelength Angstrom Exponent

Day Solar Azimuth Channel Wavelength

Hour Satellite Height Optical Depth

Minute Geolocation Quality

Second VIIRS Geolocation Quality

59

Product Requirements:VIIRS Polar Winds

Requirement: The Reformatting Toolkit shall tailor NDE-generated Polar Winds product from NetCDF4 to BUFR format for EMC.

Additional requirements for this product will be forthcoming.

60

Product Requirements: Green Vegetation Index

Requirement: The Reformatting Toolkit shall tailor NDE-generated Green Vegetation Index products from NetCDF4 to GRIB2 format for EMC.

Additional requirements for this product will be forthcoming.

61

Interface Requirements Requirement: The Reformatting Toolkit shall use the thinned

CrIS radiances (~300 channels) files from NUCAPS as an input for generating the CrIS radiance BUFR files. These files shall be in compliant NetCDF4 and shall already contain the thinned SDR and geolocation information.

Requirement: The Reformatting Toolkit shall use the NPOESS ATMS TDR and SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files.

Requirement: The Reformatting Toolkit shall use the NPOESS OMPS Total Column Ozone EDR files and OMPS Nadir Profile IP files tailored into NetCDF4 as an input for generating the OMPS Ozone BUFR files.

62

Interface Requirements Requirement: The Reformatting Toolkit shall use the NPOESS

VIIRS radiance SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the VIIRS radiance BUFR files.» Requirement: The Reformatting Toolkit shall receive the NPOESS

VIIRS IP Cloud Mask Product to obtain the cloud and land mask for the VIIRS radiance BUFR files.

Requirement: The Reformatting Toolkit shall use the NPOESS Aerosol Optical Thickness EDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the Aerosol Optical Thickness EDR BUFR files.

Requirement: The Reformatting Toolkit shall use the NPOESS SST EDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the SST BUFR files.

63

Requirements Summary

The Reformatting Toolkit developers have met with NDE and all the users of the original Phase I NDE tailored products.

All Reformatting Toolkit project, functional, and product requirements have been captured and documented in the Reformatting Toolkit RAD.

As development continues, the detailed product requirements shall continue to be refined.

64

Review Outline


65

Software Architecture

Presented by

Thomas KingPSGS

66

Reformatting Toolkit Architecture

Hardware Environment» Development and Unit testing» Test Product Distribution» System Testing and Integration» Production

Data Files» File Formats» Input Files» Static/Ancillary Files» Output Files» Run Files

Software Description» External Interfaces» System Level» Unit Level

67

Reformatting Toolkit Development Hardware

Reformatting Toolkit Development Hardware - Unit tests, and CrIS/ATMS simulation data generation will be conducted on the IASI development machine at NSOF. This machine is maintained under the ESPC maintenance contract.» IBM P570 (AIX 5.3)» 6 TB disk space» 16 CPU» 2 GB memory/CPU» IBM XL Version 7.0 C/C++» IBM XL Version 10.1 Fortran 77/90

68

Reformatting Toolkit Development Hardware

Additional Development Hardware - Reformatting Toolkit testing may be conducted at STAR on AIX. This machine was purchased with ground systems money. It will be located at NCWCP, and maintained by STAR IT.» IBM P570» 3 TB disk space» 16 CPU» 2 GB memory/CPU

Configuration management of Reformatting Toolkit code will be conducted in the STAR Collaborative Environment on STAR Linux hardware running IBM Clear Case.

69

Reformatting Toolkit Test Product Distribution

STAR Data Server - Reformatting Toolkit test products will be available on a distribution server at STAR (ftp2.orbit.nesdis.noaa.gov). » Linux» 3.2 TB disk space» Access via anonymous ftp» Products available on a near real time basis

This server will make available test BUFR and GRIB2 files containing simulated data, BUFR tables, and any additional resources.

The first finalized version of a CrIS test BUFR file was made available to EMC (Jeff Ator and Dennis Keyser) April 15, 2009.

CrIS test BUFR files were made available to users on a near real time basis on June 19, 2009.

Yong Chen at the JCSDA began accessing the CrIS test BUFR files on July 14, 2009. A BUFR table and reader subroutine were also provided.

70

Reformatting Toolkit Integration and

Production Hardware NDE SADIE – The Reformatting Toolkit system

testing and integration will be conducted on the SADIE integration platform working with NDE integration personnel. This hardware is located at NSOF and is maintained under the ESPC maintenance contract. » IBM P561 (AIX 5.3)» 50 TB disk space» 16 CPUs» 2 GB/CPU» IBM XL Version 9.0 C/C++» IBM XL Version 11.1 Fortran 77/90

71

Reformatting Toolkit Integration and

Production Hardware NDE Test/Production Hardware - After successful

system tests, our understanding is that NDE plans to check the code into configuration management and then promote it to the Test/Production machine. This machine will be located at NSOF. It has not yet been acquired.» IBM P570 (AIX 5.3)» TBD Disk space» TBD CPUs» TBD GB/CPU

72

NetCDF4 File Format Information

NetCDF (Network Common Data Form) is a machine independent binary format that was developed by UCAR (University Corporation for Atmospheric Research).

The latest version of NetCDF is version 4. Version 4 is built on top of HDF version 5.

Unlike BUFR and GRIB, it was not designed with a particular meteorological application so its records are not designed to assume any particular geographic reference.

Data are stored as arrays making it useful for storing sequentially organized data such as satellite data.

73

NetCDF4 File Format Information

The CF (Climate and Forecast) Convention defines metadata that are included in the same file as the data (thus making the file "self-describing"), that provide a definitive description of what the data in each variable represents, and of the spatial and temporal properties of the data. Examples of CF metadata include specification of: » Coordinate systems information needed to locate data in space and

time» Standard names for quantities to determine whether data from

different sources are comparable» Information about grids, such as grid cell bounds and cell averaging

methods

The actual structure of data storage in NetCDF is not known or visible to the user. The user must access the data solely through the NetCDF APIs.

74

BUFR File Format Information

BUFR (Binary Universal Form for the Representation of meteorological data ) is a machine dependent form of binary file.

The format is standardized by the World Meteorological Organization (WMO) Commission for Basic Systems (BUFR FM 94).

The current standard is version 4, although version 3 is still an accepted version for operations.

It is used primarily to store station data and was designed as a format for transmission.

75


Each BUFR product file is constructed from a static table that contains BUFR descriptors.

BUFR “descriptors” are used to indicate the exact meaning and storage structure of the data values.

A set of WMO master BUFR tables defines all standard descriptors. This makes the format self-describing.

For example, radiance can be described as:» Table B descriptor: 0-14-045 » Description: channel radiance » Units: W m-2 sr-1 cm-1» Scale, Offset, Bit storage: 0 0 11 » Mnemonic: CHRAD

If a particular type of data cannot be described using an existing descriptor, a new descriptor may be proposed. Doing so requires our NOAA representative to WMO (currently Jeff Ator) to present the change at the bi-annual WMO meeting.

76


A BUFR message is composed of six sections, numbered zero through five.» SECTION 0 - Indicator Section contains “BUFR”, length of message, edition

number.» SECTION 1 - Identification Section contains length of section and identification

of message.» SECTION 2 - Optional Section; if used, it may contain arbitrary data in any

form wished for by the creator of the message (this is only advisable for local use).

» SECTION 3 - Data Description Section contains a sequence of so-called descriptors that define the form and contents of the BUFR data product.

» SECTION 4 - Data Section; a bit-stream containing the message's core data and meta-data values as laid out by Section 3.

» SECTION 5 - End Section “7777”.

A given BUFR file may contain multiple BUFR messages. The table determines only the structure of the message, not the number of messages or the size of the BUFR records; that is determined by the user’s software.

77

GRIB2 File Format Information

GRIB (Gridded Binary) is a binary machine independent format used and generated (as forecast output) by NWP.

GRIB (like BUFR) is standardized by the WMO’s Commission for Basic Systems (GRIB FM 92-IX, described in WMO Manual on Codes No.306).

The latest version of GRIB is Edition 2 although Edition 1 is still widely used.

It is designed for storing gridded data.

78


A GRIB file can contain many different variables on different grids.

Like BUFR, these names are standardized in tables by WMO so the file is self-describing.

For example, if you want write temperature, in Octet 11 of section 4 of the GRIB2 you set:» GRIB Code for Temperature: 0» Abbrev: TMP» Units: Kelvin

79


A GRIB2 file contains the following sections:» SECTION 0 - Indicator Section contains “GRIB”, Discipline, GRIB Edition number,

length of message.» SECTION 1 - Identification Section contains length of section, section number,

message characteristics. » SECTION 2 - (Local Use Section) – optional; length of section, section number,

additional local use info.» SECTION 3 - Grid Definition Section contains definition of grid and geometry of data.» SECTION 4 - Product Definition Section (repeated) contains description of the

nature of the data. » SECTION 5 - Data Representation Section (repeated) contains description of how

the data values are represented.» SECTION 6 - Bit-map Section (repeated) contains an indication of presence or

absence of data at each grid point.» SECTION 7 - Data Section contains the data.» SECTION 8 - End Section “7777”.

Sequences of GRIB sections 2 to 7, 3 to 7, or sections 4 to 7 may be repeated within a single GRIB message.

80

Reformatting Toolkit Data

The tables on the following slides show all the Reformatting Toolkit input and output. The input data files are those required by the Reformatting Toolkit software to produce the Phase I tailored products.

All input data must be CF compliant NetCDF4 format.

Ancillary/Static files, such as the BUFR tables are listed.» They will be delivered as part of the DAP» Described more thoroughly in the CDR

All output will be in either BUFR or GRIB2 format.

81

Reformatting Toolkit Input Data Files

File Format Source Description PurposeCrIS Thinned SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned ~300 channel radiances

for all FOVs.CrIS BUFR

ATMS TDR NetCDF4 NDE NPOESS ATMS full resolution file tailored into NetCDF4 from the NPOESS HDF5.

ATMS BUFR

ATMS SDR NetCDF4 NDE NPOESS ATMS full resolution file tailored into NetCDF4 from the NPOESS HDF5.

ATMS BUFR

ATMS SDR/TDR Geo NetCDF4 NDE NPOESS ATMS Geolocation file that is associated with the ATMS SDR and TDR. This is the same file.

ATMS BUFR

VIIRS M-Band 01 SDR NetCDF4 NDE NPOESS VIIRS Moderate resolution band SDR for channel 1.

VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR

82


File Format Source Description PurposeVIIRS M-Band 08 SDR NetCDF4 NDE NPOESS VIIRS Moderate resolution band SDR for

channel 8.VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR


VIIRS BUFR

VIIRS I-Band 01 SDR NetCDF4 NDE NPOESS VIIRS Imagery resolution band SDR for channel 1.

VIIRS BUFR

VIIRS I-Band 02 SDR NetCDF4 NDE NPOESS VIIRS Moderate resolution band SDR for channel 2.

VIIRS BUFR

83


File Format Source Description PurposeVIIRS I-Band 03 SDR NetCDF4 NDE NPOESS VIIRS Moderate resolution band SDR for

channel 3.VIIRS BUFR


VIIRS BUFR


VIIRS BUFR

VIIRS DNB SDR NetCDF4 NDE NPOESS VIIRS Day/Night Band SDR. VIIRS BUFR

VIIRS M-Band SDR Geo NetCDF4 NDE VIIRS Moderate resolution band SDR Geo file. VIIRS BUFR

VIIRS I-Band SDR Geo NetCDF4 NDE VIIRS Imagery resolution band SDR Geo file. VIIRS BUFR

VIIRS DNB SDR Geo NetCDF4 NDE VIIRS Day/Night Band SDR Geo file. VIIRS BUFR

VIIRS Cloud Mask IP NetCDF4 NetCDF4 NDE VIIRS granule files containing cloud data. VIIRS BUFR

VIIRS Cloud Mask IP Geo NetCDF4 NetCDF4 NDE VIIRS granule files containing the geolocation information for the cloud product.

VIIRS BUFR

OMPS Nadir IP NetCDF4 NDE NPOESS OMPS Nadir Profile ozone Intermediate Product file tailored into NetCDF4. Geolocation information is currently in the file. NGAS documentation indicates that much of the content of this product is TBD.

OMPS BUFR

OMPS Total Column EDR NetCDF4 NDE NPOESS OMPS Total Column ozone EDR file tailored into NetCDF4.

OMPS BUFR

OMPS Total Column SDR Geo NetCDF4 NDE NPOESS OMPS Total Column ozone SDR Geo file tailored into NetCDF4. This same Geo file is the used with the OMPS Total Column EDR.

OMPS BUFR

84


File Format Source Description PurposeAerosol Optical Thickness EDR NetCDF4 NDE NPOESS AOT EDR file tailored into NetCDF4. AOT BUFR

Aerosol Optical Thickness EDR Geo NetCDF4 NDE NPOESS AOT EDR Geo file tailored into NetCDF4. AOT BUFR

Sea Surface Temperature EDR NetCDF4 NDE NPOESS SST EDR file tailored into NetCDF4. SST BUFR

Sea Surface Temperature EDR Geo NetCDF4 NDE NPOESS Moderate Resolution Terrain-Corrected Geolocation file tailored into NetCDF4. This is to be used with the NPOESS SST EDR.

SST BUFR

Polar Winds EDR (New) NetCDF4 Jeff Key/Jamie Daniels

The Polar winds EDR produced by Jeff Key and Jamie Daniel’s product system running within NDE

BUFR

GVF EDR (New) NetCDF4 Ivan Csiszar/Hanjun Ding

The Green Vegetation Fraction EDR produced by Ivan Csiszar’s product system running within NDE.

GRIB2

85

Reformatting Toolkit Ancillary/Static Data

Files

File Format Source Description Purpose

CrIS SDR BUFR Table ASCII N4RT CrIS thinned radiances (~300 channels, all FOVs) BUFR Template

ATMS SDR BUFR Table ASCII N4RT ATMS radiances full resolution BUFR Template

VIIRS SDR BUFR Table ASCII N4RT VIIRS radiances full resolution BUFR Template

OMPS EDR BUFR Table ASCII N4RT OMPS Nadir Profile and Total Column Ozone BUFR Template

AOT EDR BUFR Table ASCII N4RT Aerosol Optical Thickness and Particle Size BUFR Template

SST EDR BUFR Table ASCII N4RT Sea Surface Temperatures, Cloud Mask BUFR Template

PW BUFR Table (New) ASCII Polar Winds Team Polar Wind products EDR BUFR Template

GVF GRIB2 Table (New) ASCII GVF Team Green Vegetation Fraction EDR GRIB2 Template

86

Reformatting Toolkit Output Data Files

File Format Description Users

CrIS SDR BUFR CrIS thinned radiances (~300 channels, all FOVs) JCSDA, NCEP, EUMETSAT, UKMet, ECMWF

ATMS SDR BUFR ATMS radiances full resolution JCSDA, NCEP

VIIRS M-band SDR BUFR VIIRS radiances moderate band JCSDA, NCEP

VIIRS I-band SDR BUFR VIIRS radiances image band JCSDA, NCEP

VIIRS DNB-band SDR BUFR VIIRS radiances day/night band JCSDA, NCEP

OMPS EDR BUFR OMPS Nadir Profile and Total Column Ozone JCSDA, NCEP

AOT EDR BUFR Aerosol Optical Thickness and Particle Size JCSDA, NCEP

SST EDR BUFR Sea Surface Temperatures, Cloud Mask JCSDA, NCEP

PW EDR (New) BUFR Polar Winds JCSDA, NCEP

GVF EDR (New) GRIB2 Green Vegetation Fraction JCSDA, NCEP

87

Reformatting Toolkit Run Data Files

File Format Source Description PurposePCF ASCII NUCAPS The PCF file containing all the required input

parameters for the N4RT driver script.Driver input

N4RT Run Log ASCII N4RT The N4RT log file containing all the standard error and standard output from a given run.

Diagnostic

N4RT Main Resource ASCII N4RT The internal control file required by the N4RT executable main program.

Executable input

PSF ASCII N4RT The Process Status File containing only the successfully generated output files.

Driver output list

88

Reformatting Toolkit Software: External Interfaces

The Reformatting Toolkit system will be run via the execution of a single driver script that will be invoked, monitored, and managed by the NDE DHS Product Generation Manager.

Execution of the script will be event-driven (i.e. the arrival of a required input file whose type is predefined in the Reformatting Toolkit production rules given to NDE).

An invocation of Reformatting Toolkit by the NDE DHS will be used to perform a single file-to-file (e.g. granule-to-granule) translation (as opposed to reformatting many files as once).

Our understanding is that NDE plans to run the driver script in a working directory. All Reformatting Toolkit output will be produced in this same working directory.

The driver script will require an input PCF file containing parameters such as:» The input and output file names» Input and output product type IDs (e.g. ATMS radiances)» The type of conversion required (e.g. NetCDF4 to BUFR)» BUFR or GRIB2 tables

89

Reformatting Toolkit Software: External Interfaces

The driver script will run, parse the PCF content, run the compiled conversion code, handle program output and errors, direct required error codes to the DHS via an output log (and through the driver script’s return code), and generate a PSF.

If there are errors, our understanding is that NDE plans to save the contents of the run in a forensics repository.

Our understanding is that NDE plans to manage and direct error status to the operators from the DHS system.

Our understanding is that NDE plans to manage all distribution through the NDE DDS and the short term storage of tailored data on the SAN.

90

Reformatting Toolkit External

Interfaces to NDE

Reformatting Toolkit Driver

Script

SAN

NDE Product Generation Manager

Reformatting Toolkit External Interfaces

Product Generation

Specifications

Working Directory

Systems Configuration

s

Forensics Repository

Input Files &

PCF

InvocationProcess Req.

Rule SetsOutput Files &

PSF

BUFR & GRIB2

Output Files

PSF (N4RT output)

Return Code

Working Directory Output

Input Files (NetCDF4)

NDE DDS

PCF (N4RT input)DAP Specifications

Data AreasConfigurations InfoN4RT SystemNDE Production Manager

NDE DHS Boundary

Input Files (NetCDF4)

91

Reformatting Toolkit Software: System Level

The Reformatting Toolkit driver script will be a single Perl script that acts as a wrapper for the compiled Reformatting Toolkit code.

» There will be no hard coded paths in the script. All needed information regarding locations of files will come through the PCF.

» All system calls have their return values checked so the exits are graceful and informative.

» All standard out and standard error will be directed to a single log file that can be read by NDE for obtaining any error or warning messages.

» The driver script will translate the low-level program errors into the high-level numerical error codes expected by the PGM.

» The Perl script will generate an internal control file for the main Reformatting Toolkit program.

92

Reformatting Toolkit Software: System Level

The PCF content (nominal mode):» Input NetCDF4 file(s)» Output BUFR file» Output GRIB2 file» Direction of conversion (eg. NC to BF)» Input and output product types» Input/Output BUFR table» Input/Output GRIB2 table

The PCF content (test mode): » Input BUFR or input GRIB2» Output NetCDF4 file» Direction of conversion (e.g. BF to NC)» Input and output product types» NC templates if (NetCDF4 is an output type)

93

Reformatting Toolkit System

Level Data FlowReformatting Toolkit System Level Data Flow

PCF Return Value to PGM

Execution from PGM

N4RT Driver Script

Working directory

PSF

Working directory

N4RT MainConverter N4RT log

BUFR file

Resource

GRIB2 file

Working directory

NetCDF4 NC Template

BF or GB table

NetCDF4

BUFR/GRIB2

Working directory

Nominal ModeTest Mode

94

Reformatting Toolkit Software: Unit Level

The Reformatting Toolkit main program will be a single main C++ program containing all the predefined data structures required by the code. Lower-levels will be in Fortran 90. Why this approach?» C++ main program will give the system flexibility for having to plug into future

types of architectures.» APIs for BUFR are only Fortran 77.» Much of the code being reused is already in Fortran 90.» Many users tend to request readers in Fortran 90.

In the Reformatting Toolkit main program, all data structures will be declared and a series of cases statements will direct the program flow based on the type of input files and the direction of conversion.

The Reformatting Toolkit subroutines at the next level down (and at all subsequent levels) will be in Fortran 90. This next level will manage:» Allocation, initialization, deallocation of data structures. » They are designed for overseeing specific product definitions and conversions.

95

Reformatting Toolkit Software: Unit Level

The lowest-level Reformatting Toolkit subroutines:» Perform the actual reading and writing of the specific file types.» Directly call the library APIs for BUFR, GRIB2, and NetCDF4.

General Reformatting Toolkit compiled code characteristics:» The status of all functions are checked to allow for graceful and

informative exits.» No paths are hard coded in the compiled code.» The are no system calls from within the compiled code.» All code will be compiled as 64 bit to utilize the IBM architecture.» All code will be “mostly statically” linked (i.e. only system libraries like

libc.a and the 64-bit UNIX kernel will be dynamically linked).

96

Reformatting Toolkit Unit

Level Data FlowReformatting Toolkit UNIT Level Data Flow

Prod N - NC2BF Prod N - BF2NCProd N - NC2GB Prod N - GB2NC

Allocate

Deallocate

Initialize

Prod N - Read BF

Prod N – Write NC

Allocate

Deallocate

Initialize

Prod N - Read NC

Prod N - Write GB

Allocate

Deallocate

Initialize

Prod N - Read GB

Prod N - Write NC

Allocate

Deallocate

Initialize

Prod N - Read NC

Prod N - Write BF

N4RT resource N4RT log file

BUFR GRIB2 NetCDF4 NetCDF4

N4RT Main

Nominal ModeTest Mode

97

Reformatting Toolkit Software: Libraries

BUFR Library » NCEP BUFRLIB library (available on the NCEP website)» Code is all Fortran 77» Compiled as 64 bit

GRIB2 Library» NCEP GRIB2 Encoder/Decoder library (available on the NCEP website)» Code is available in C and Fortran 90» Compiled as 64 bit

NetCDF4 Library version 4.0» Available from Unidata website» Compiled as 64 bit

Reformatting Toolkit developers will coordinate with NDE and the product teams to make sure that all parties are using compatible libraries.

98

Software Architecture Summary

Reformatting Toolkit software development, testing, and operation will be conducted on comparable IBM hardware at NSOF and STAR.

The Reformatting Toolkit code will run as a stand-alone unit within the NDE DHS.

The code will be modular to allow for easy reuse of code and expansion to accommodate new products.

The code will use only the official releases and NDE compatible versions of the NetCDF4, BUFR, and GRIB2 libraries.

A Reformatting Toolkit Prototype DAP shall be delivered at the end of September 2009.

99

Review Outline


100

Quality Assurance

Presented by

Thomas KingPSGS

101

Quality AssuranceBackground

STAR is using the Capability Maturity Model Integrated (CMMI) to improve processes and practices for development and the transfer of research to operations.

The STAR Enterprise Life Cycle (EPL) merges the CMMI and SPSRB standards.

IASI and NUCAPS are both pathfinder projects (CMMI Level 3). The IASI system has been transitioned to operations successfully.

The Reformatting Toolkit algorithm development will follow the same process but tailored to the scale of the project.

102

The Reformatting Toolkit Preliminary Design Review (April 2009) » Will present the initial draft of the requirements and discuss a proposed design.» An Reformatting Toolkit Requirements Allocation Document (RAD) has been made

available to Reformatting Toolkit stakeholders. It will be updated throughout the lifecycle of the project.

The Reformatting Toolkit Critical Design Review (September 2009)» To finalize requirements and to verify that the chosen design is able to meet those

requirements.

An Reformatting Toolkit Test Readiness Review (April 2010) » Will present the unit test plan to demonstrate that the toolkit is ready to be run in the

Test Environment.

A Code Unit Test Review (May 2010) » Will be conducted to ensure that the Reformatting Toolkit software is able to fulfill the

functional software requirements.

The Reformatting Toolkit System Readiness Review (January 2011) » Has requested to be waived because Reformatting Toolkit will be run within the NDE

system.

Quality Assurance – ProjectImplemented Tailored STAR EPL

Reviews

103

CM Tool (IBM Rational ClearCase, Version 7.0 ) » Has been purchased and implemented in the Collaborative

Environment.

CM personnel have been identified.

CM training:» Administrator training completed. » Developers will be trained by the CM administrator.

We will be using a modified version of the CM plan developed for GOES-R framework being developed at STAR.

Configuration Management (CM)

104

STAR Coding Standards

Coding standards guidelines and quick references are available.

Provide a common list of abbreviations.

Adhere to the standards throughout the development life cycle.

Have checklists available for developers to keep track of the delivery status of the code.

105

Quality Assurance – Software

A prototype version of the Toolkit will be tested on the NDE hardware as part of the NDE Build Content Reviews during the fall of 2009.

All code development is being conducted on a platform that is nearly identical to the test and production target platforms using the same compilers and operating system.

Only the official releases of the NCEP BUFRLIB, GRIB2, and NetCDF4 libraries will be used in the software.

STAR code checking tools will be used to minimize coding bugs and to ensure that Reformatting Toolkit code meets STAR coding standards.

The status of all system calls and intrinsic functions are checked.

Unit tests will have the Reformatting Toolkit generate BUFR and GRIB2 products from NetCDF4 files. These BUFR and GRIB2 files will be directed back into the Reformatting Toolkit to generate new NetCDF4 files.» The contents of the new and original NetCDF4 files will be compared to demonstrate that what went

in matches with what came back out.» Customers will have access to test data products to verify that values appear reasonable and that

precision is not being lost in the format conversions.

106

Quality Assurance – Software

An official (tailored) DAP will be delivered:» All Reformatting Toolkit code and system files» Test plans» Test data sets» Error messaging/handling» PCF format» Production rules» Product file specifications» Data flow diagrams» Estimates of resource usage» ATBD has been waived

– However, all modifications to data will be identified and document» Delivery memo

107

Quality Assurance – Products

Reformatting Toolkit developers will work with EMC, NRL, FNMOC, GMAO, EUMETSAT, and UK Met users to ensure that the proper BUFR descriptors are used in the BUFR and GRIB2 tables and that, whenever possible, they are WMO approved (as opposed to local). » Chosen descriptors must work with NCEP BUFRLIB.

Reformatting Toolkit developers will work with EMC, heritage product developers, and NPOESS operational algorithm teams to ensure consistency with heritage products with respect to format and content.

Reformatting Toolkit developers will make BUFR tables and test products (BUFR and GRIB2) available to users before the operational products are made available. This will allow for preliminary product content validation.» CrIS and ATMS are currently being simulated at STAR.» For the other products such as VIIRS, we are working with the IPO (Joe Zajic),

the NPP Test Data Working Group (TDWG), and NDE to obtain simulated products.

108

Current Status of BUFR Table

Development

File Table Status Test Data AvailableCrIS SDR BUFR Table

Finished Yes (July 2009)

ATMS SDR BUFR Table

Approved by NCEP and WMO CGMS committee, but awaiting WMO codes group approval

No (October 2009)

VIIRS SDR BUFR Table

Released to NCEP and is currently under review No

OMPS EDR BUFR Table

Table development is currently in progress No

AOT EDR BUFR Table


SST EDR BUFR Table


109

Quality Assurance – Archive and Maintenance

Archive Plan» Reformatting toolkit will be integrated into NDE system and made

available to CLASS by NDE» Product archive requirements are addressed within product

development projects– NPOESS program office works with CLASS to archive xDRs– NOAA Unique Product (NUP) projects work with CLASS as

required

Long Term Maintenance Plan» The reformatting toolkit will be maintained by the OSDPD staff» STAR system developers will be available

110

Quality Assurance – Documentation and

Metadata Documentation/Metadata Plan

» The Documentation will include those sections of the SPRSB documents that are decided to be the responsibility of the project teams. The decisions regarding who is responsible for writing the different parts of these documents are being made in a series of ongoing meetings involving the Standards Working Group, NDE, and the STAR Product Teams.

» When agreement has been made, the Tailoring project’s requirements will be updated to include generation of the necessary portions of these documents.

» Given the current trajectory of this group’s work, we anticipate involvement in the creation of the following documents:– SMM (System Maintenance Manual)– UM (User’s Manual)– DDD (Data Description Document)– SDD (System Description Document)– No MDD (Metadata Description Document)– No OM (Operations Manual)

» Additional CMMI documents:– RAD (Requirements Allocation Document)

» NDE Documents:– DAP (Testing information is contained within here)

111

Quality AssuranceSummary

Quality assurance plan will consist of: » Project reviews at which stakeholders are encouraged to participate.» Ongoing interaction with customers, heritage product developers,

operations, NDE, and the SPSRB.» Adhering to STAR/NDE software standards and use of standard

libraries only.» Software unit tests shall be presented in the TRR.» Documentation of the Reformatting Toolkit code operation, production

rules, and software tests will be in the DAP.» Documentation of requirements will be in the Reformatting Toolkit

RAD.» Early release of BUFR and GRIB2 products, tables, and additional

resources will allow for customer validation of products.

112

Review Outline


113

Risks and Actions

Presented by


114

Risk and Actions

The status of risks from the PDR Report were reported earlier in this CDR, but will be summarized here.

The status of any new risks since the PDR will be reported here as well.

115

Risk Summary Risk 1: NPOESS product formats and content are still

changing.» Mitigation: Work and maintain contact with NPOESS DFWG and

customers.

Risk 2: Roles and responsibilities for SPSRB documents.» Mitigation: Work with Maurice McHugh and SPSRB Standard Group.

Risk 3: Variations in platforms and compilers.» Mitigation: Delivery of prototypes and coordination with NDE on

system tests.

Risk 4: Data format translation impacts (unit conversion & precision)» Mitigation: Document any translation of data in the DAP.

116

Risk Summary

Risk 6: GRIB2 functionality not demonstrated.» Mitigation: Test GRIB2 capability by generating NDVI GRIB2 file.

Risk 8: Need to verify with EMC management regarding NDVI and Snow Cover. » Mitigation: Tom Schott meet with Steve Lord.

Risk 9: User access to spectral response and calibration. » Mitigation: IPO to work with NPOESS contractor.

117

Risk Summary

Risk 11: Effort imposed on product teams to generate NetCDF4.» Mitigation: Issue is for NDE and product teams to resolve.

Risk 12: Schedule risk associated with product teams having to convert formats.» Mitigation: Issue is for NDE and product teams to resolve.

118

Risk and ActionsSummary

There are currently 9 risks identified for the Reformatting Toolkit project. These carry over from the PDR.

There are no new risks at this time.

We believe that 3 of these can be closed.

6 risks remain open.

An additional risk (risk #8) may be closed depending on status.

119

Review Outline


120

Summary and Conclusions

Presented by

Thomas KingNOAA/NESDIS/STAR

121

The Project Mission and Schedule has been reviewed.

The PDR Report has been reviewed.

The Project Requirements have been reviewed.

Software architecture, hardware, data, and interfaces have been reviewed.

Plans for quality assurance have been reviewed.

Risks and Actions have been reviewed.

Review Objectives Have Been Addressed

122

Current Status

The CrIS BUFR table is finished and test BUFR data are available on the STAR ftp2 server via anonymous ftp.

The ATMS BUFR table has been reviewed by EMC (Dennis Keyser & Jeff Ator), EUMETSAT (Simon Elliott), ECMWF(Milan Dragosavac), UKMet (Nigel Atkinson) and the WMO CGMS science committee. It is awaiting WMO approval.

VIIRS radiance, SST, and AOT tables have been delivered to EMC and are currently under review.

OMPS BUFR table is in progress.

A prototype of the code has been created. There is an NDE compatible Perl driver (with PCF, PSF, and log generation). » A Fortran executable can currently convert CrIS between NetCDF4 and BUFR (in both

directions). » ATMS read/write BUFR capability is being added at this time. We anticipate delivery of a

prototype to NDE at the end of September 2009.

Reformatting Toolkit developers will be working with Joe Zajic (IPO) to obtain access to simulated VIIRS products at NSOF.

123

Next Steps

Code development phase is ongoing» Delivery of prototype DAP to NDE» Delivery of ATMS test BUFR files to EMC» Obtain WMO approval of ATMS, VIIRS, OMPS, SST,

and AOT BUFR tables» Obtain VIIRS test data

Test Readiness Review (April 2010)

124

Open Discussion

The review is now open for free discussion

125

Backup Slides

126

Project BackgroundTerminology

Raw Data Records (RDRs)

Sensor Data Records (SDRs)

Temperature Data Records (TDRs)

Environmental Data Records (EDRs)

Intermediate Products (IPs)

127


Raw Data Records (RDRs)» Full resolution, digital sensor data, time-referenced and

locatable in earth coordinates with absolute radiometric and geometric calibration coefficients appended, but not applied, to the data.

128


Sensor Data Records (SDRs)» Data record produced when an algorithm is used to

convert RDRs to geolocated, calibrated detected fluxes with associated ephemeris data. Calibration, ephemeris, and any other ancillary data necessary to convert the sensor units back to sensor raw data (counts) are included.

129


Temperature Data Records (TDRs)» Calibrated antenna temperature records from a

microwave instrument such as ATMS.

Environmental Data Records (EDRs)» Data records produced when an algorithm is used to

convert SDRs to geophysical parameters (including ancillary parameters, e.g., cloud clear radiation, etc.).

130


Intermediate Products (IPs)» There are 2 types of IPS– DIPs: These are “Delivered IPs” meaning that these

files will be made available to NPOESS customers via the IDPS.– RIPs: These are “Retained IPs” meaning that these

files will be retained within the IDPS and to released to customers.

prepared by: tom king 1 , zhaohui cheng 1 , larisa koval 1 , and yi song 2 , 1 psgs, 2 imsg

Documents

npoess mission

nde products

nde product training

noaaunique products

product enhancement

data manipulations

output product formats

environmental products