Download - Prepared By: Tom King 1 , Yi Song 1 , 1 Kexin Zhang, Larisa Koval 1 , and Walter Wolf 2 ,
1
NetCDF4 Reformatting Toolkit (N4RT): BUFR and GRIB2
Tailoring Test Readiness Review (TRR) for
Phase 1 SDR ProductsOctober 25, 2011
Prepared By: Tom King1, Yi Song1,1Kexin Zhang, Larisa Koval1,
and Walter Wolf2, 1Riverside, 2STAR
2
Purpose of TRR/CTR· Review the status of all the open issues/risks
· Review Project Requirements
· Describe the Software Architecture
· Describe the tests for the software units and show the test results
· Establish the contents of the Delivered Algorithm Package
· Identify any new issues/risks
3
Review AgendaSection Time Presenter
Introduction 9:00-9:10 Walter Wolf
CDR Review Report/Risks and Actions 9:10-9:25 Thomas King
Requirements 9:25-9:40 Thomas King
Quality Assurance 9:40-9:50 Thomas King
Software Architecture 9:50-10:10 Thomas King
Unit Tests 10:10-10:50 Yi Song
Delivered Algorithm Package 10:50-11:00 Thomas King
Risks and Actions Summary 11:00-11:15 Thomas King
Summary and Conclusions 11:15-11:20 Walter Wolf
4
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
5
Section 1 – Introduction
Presented by
Walter WolfNOAA/NESDIS/STAR
6
Introduction· Project Background
» IJPS» NPP/JPSS» NDE
· Project Objectives
· Integrated Product Team
· Project Plan
· Entry and Exit Criteria
7
JTA and IJPS
• JTA – Joint Transition Activities» JTA is a replacement of the Initial Joint Polar-Orbiting Operational
Satellite System (IJPS) agreement and is designed to cover only the NPP mission.
» IJPS started with NOAA-N and covers the MetOp series. JTA and IJPS are cooperative efforts between NOAA and EUMETSAT to provide and improve the operational meteorological and environmental forecasting and global climate monitoring services worldwide.
• The JPSS J1 and J3 data availability will be covered by the Joint Polar-Orbiting Operational Satellite System (JPS) agreement.
8
NPP/JPSS• NPP and JPSS, a joint NOAA/NASA effort, is the next
series of polar-orbiting satellites dedicated to among other things, operational meteorology. The objective of the JPSS mission is to ensure continuity, improvement and availability of operational observations from an afternoon polar orbit (13:30 pm).
• Meteorological/Climatological Instrument packages on NPP/JPSS:» CrIS, ATMS, VIIRS, OMPS, CERES
• NPP is the first of 3 missions with a launch date of October 28, 2011.
9
Project BackgroundNDE
· Disseminate NPP/JPSS Data Records to customers.
· Generate and disseminate tailored NPP/JPSS Data Records (versions of NPP/JPSS Data Records in previously agreed alternative formats and views).
· Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPP/JPSS Data Records).
· Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving.
· Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA.
· Provide software for NPP/JPSS Data Record format translation and other data manipulations.
10
· To build a software package that will tailor JPSS and NDE products from NetCDF4 into BUFR and GRIB2 formats in support of NDE’s overall tailoring efforts.
· The NetCDF4 Reformatting Toolkit (N4RT) must be designed so it can easily be modified/expanded to incorporate the tailoring of new products.» Flexible» Extendable
· The software must be able run within the NDE system architecture and operate within the NDE functional guidelines.
· Output product formats and content must meet the needs of NOAA customers.
N4RT Project Objectives
11
· Phase 1 SDR» CrIS Radiances (BUFR)» ATMS Radiances (BUFR)» VIIRS Radiances (BUFR)
· Phase 1 EDR» Sea Surface Temperature (BUFR)» Aerosol Optical Thickness (BUFR)» Nadir Profile and Total Column Ozone (BUFR)
· Phase 2 EDR» Polar Winds (BUFR)» Green Vegetation Fraction (GRIB2)
· Phase 3 EDR» OMPS Limb Profiles (BUFR)» AVHRR, GOES, VIIRS Cloud Products (GRIB2)
N4RT Project Objectives
Phase 1 Products
12
Integrated Product Team
· IPT Lead: Walter Wolf (STAR)
· IPT Backup Lead: AK Sharma (OSPO)
· NESDIS team:» STAR: Walter Wolf, Jaime Daniels, Yi Song, Thomas King, Kexin Zhang, Larisa Koval» NDE: Jim Silva, Geof Goodrum, Richard Sikorski, Kevin Berberich» OSPO: Dave Benner, AK Sharma, Ricky Irving» OSD: Tom Schott
· User team» Lead: Jim Heil (NWS), Stephen Lord (NWS /NCEP/EMC), John Derber
(NWS/NCEP/EMC), Jeff Ator (NWS/NCEP/NCO), Lars Peter-Riishojgaard and Jim Yoe (JCSDA), Simon Elliott (EUMETSAT), Tony McNally (ECMWF), Fiona Hilton (UK-Met)
» Others: International NWP users, NWP FOs, Climate Users
· Product Oversight Panel: ZPOP, EPOP, ICAPOP, CAL/NAVPOP
13
Project Stakeholders
· NOAA National Weather Service» Weather Forecast Offices» National Center for Environmental Prediction
· Department of Defense» NRL» FNMOC» AFWA
· Global NWP» EUMETSAT» ECMWF» UK Met» Meteo France» CMC» JMA» BOM
14
Plan of Operations – Phase 1
· Year 1 – Design and Development» Evaluate the requirements; work with NDE» Discuss with the current developers of similar translators to determine what is
required in their output files» Design the NetCDF4 reformatting toolkit; distribute to OSPO and NDE for review» Conduct PDR» Create generic NetCDF4 readers and writers» Develop BUFR tables and GRIB formats with the product teams for Phase 1
products» Work with NDE to determine the interface between the SDR and EDR NPP
products and the reformatter» Conduct CDR
15
Plan of Operations – Phase 1
· Year 2 –Transition to Pre-Operations of Phase 1 Products» Set up infrastructure to implement the readers and writers for the data formats;
work with NDE to determine the interface to the data handling system; make available to OSPO for review
» Implement BUFR tables and GRIB formats for the Phase 1 products on the NDE hardware; work with NDE and OSPO to evaluate the implementation
» Conduct Test Readiness Review for Phase 1 products» Transition Phase 1 product reformatters to pre-operational system on the NDE
hardware» Test system within the NDE environment» Prepare Documentation» Conduct Code Review for Phase 1 products
· Year 3 – Transition to Operations of Phase 1 Products» Evaluate with NDE and OSPO the implementation of the Reformatter within the
NDE data handling system» Conduct Algorithm Readiness Reviews (separate reviews for Phase 1 SDR and
EDR products)» Transition pre-operational Phase 1 product reformatting system to operations
16
Plan of Operations – Phase 2
· Year 1 – Design and Development» Evaluate the requirements; work with NDE» Discuss with the current developers of similar translators to determine what is
required in their output files» Develop BUFR tables and GRIB formats with the product teams for Phase 2 products» Conduct CDR» Implement BUFR tables and GRIB formats for the Phase 2 products on the NDE
hardware; work with NDE and OSPO to evaluate the implementation
17
Plan of Operations – Phase 2
· Year 2 –Transition to Pre-Operations of Phase 2 Products» Conduct Test Readiness Review for Phase 2 products» Transition Phase 2 product reformatters to pre-operational system on the NDE
hardware» Test system within the NDE environment» Prepare Documentation» Conduct Code Review for Phase 2 products
· Year 2 – Transition to Operations of Phase 2 Products» Evaluate with NDE and OSPO the implementation of the Reformatter within the NDE
data handling system» Conduct Algorithm Readiness Review for Phase 2 products» Transition pre-operational Phase 2 product reformatting system to operations
18
Plan of Operations – Phase 3
· Year 1 – Design and Development» Evaluate the requirements; work with OSPO» Discuss with the current developers of similar translators to determine what is required
in their output files» Develop BUFR tables and GRIB formats with the product teams for Phase 3 products
· Year 2 –Transition to Pre-Operations of Phase 3 Products» Implement BUFR tables and GRIB formats for the Phase 3 products on the OPSO
hardware; work with OSPO to evaluate the implementation» Conduct Test Readiness Review for Phase 3 products» Transition Phase 3 product reformatters to pre-operational system on the OSPO
hardware» Test system within the OPSO environment» Prepare Documentation» Conduct Code Review for Phase 3 products» Evaluate with OSPO the implementation of the Reformatter within the OSPO data
handling system» Conduct Algorithm Readiness Review for Phase 3 products» Transition pre-operational Phase 3 product reformatting system to operations
19
Project Timeline (1)
CDR 09/29/2009
PDR 04/28/2009
20
Project Timeline (2)
ARR 11/28/2011
TRR 10/25/2011
DAP 1 Delivery
11/30/2011
21
Project Timeline (3)
22
Project Timeline (4)
TRR 01/23/2012
ARR 05/30/2012
DAP 2 Delivery
06/20/2012
SCR 03/5/2012
23
Project Timeline (5)
24
Project Timeline (6)
ARR 03/10/2014
TRR 07/18/2013
DAP 3 Delivery
03/31/2014
SCR 08/26/2013
25
Project Plan – Schedule· Schedule (Milestones)
» Project begins - 7/1/08» PDR - 4/14/09» CDR - 9/14/09» Phase 1 SDR TRR/CTR - 10/25/11» Phase 1 SDR ARR - 11/15/11» Phase 1 SDR DAP Delivery - 11/30/11» Phase 1 & 2 EDR TRR/CTR - 1/23/12» Phase 1 & 2 EDR SCR - 3/5/12» Phase 1 & 2 EDR ARR - 5/30/12» Phase 1 & 2 EDR DAP Delivery - 6/4/12» Phase 3 TRR/CTR - 7/18/13» Phase 3 SCR - 8/26/13» Phase 3 ARR - 3/10/14» Phase 3 EDR DAP Delivery - 3/31/14
26
N4RT TRR Entry Criteria
· CDR Report (Review Item Disposition) http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/NetCDF4_Reformatting_Toolkit_CDR_Review_Item_Disposition_20110818.xlsx» PDR Risks and Actions» CDR Risks and Actions» CDR Actions and Comments
· Updated CDR Presentation http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_CDR_20090914.ppt
· Updated Requirements Allocation Document http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_RAD_v1.2.docx
· Review of N4RT http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/N4RT_TRR_Phase1_SDR_Products.pptx» Requirement Allocation» Quality Assurance» Software Architecture» Unit Tests and Results» Delivered Algorithm Package
27
N4RT TRR Exit Criteria
· Test Readiness Review Report»The TRR Report (TRRR), a standard artifact of the
STAR Enterprise Process Lifecycle (EPL), will be compiled after the TRR
»The report will contain:– Review Item Disposition containing all risks, actions
and comments– Updated TRR presentation
28
Review Objectives· Review the CDR Review Report (CDRR)
»Focus on actions· Review the Requirements Allocation· Review the software system architecture
»External interfaces and data flows»Dependency Tree Document
· Review the software tests and results of each software unit
· Identify risks and actions
29
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
30
Section 2 – CDR Review Report
and ActionsPresented by
Thomas KingRiverside
31
CDR Reports and Actions
· Open PDR Risks and Actions at CDR.
· CDR Risks and Actions.
· Later in the review, new risks, actions, and comments originating from the CDR, and the period since, will be presented.
32
CDR Review Report
· A CDR Report (CDRR) is produced following a project’s Critical Design Review (CDR). It is a required project artifact of the STAR Enterprise Product Lifecycle (EPL). Specifically, it is a designated artifact for the Test Readiness Review (TRR), which is a standard Technical Review in the STAR EPL process. The intended target audiences are program management, the product development team, and the TRR review team.
· A CDR Review Report is available for review in the project repository at: » http://www.star.nesdis.noaa.gov/smcd/spb/iosspdt/qadocs/N4RT/
NetCDF4_Reformatting_Toolkit_CDR_Review_Item_Disposition_20110818.xlsx
33
Open PDR Risks and Actions Since CDR
34
PDR Risk· Risk 1: JPSS product formats and content are still changing, especially
for VIIRS· Assessment: Low· Impact: May need to revise software several times during development
to adjust to new formats, names, and types.· Likelihood: High· Mitigation:
» Work through the Data Format Working Group to obtain information on format and algorithm updates. Monitor the latest copies of the Common Data Format Control Books (CDFCB) for any updates. Maintain contact with customers to inform them of any upstream product changes. Make the code design flexible so that changes in the upstream products translate into the minimum amount code revision.
· Status: Closed. We recommend closing this as CDFCB formats are now frozen and the reformatter is able to read all necessary data from the P72 datasets to which NDE subscribes.
35
PDR Risk· Risk 2: The roles and responsibilities regarding who shall generate
the set of required SPSRB documents for NDE has not yet been decided.
· Assessment: Low· Impact: Difficult to budget time needed for the team to generate
documentation.· Likelihood: Moderate· Mitigation: This issue, and that of document content, is being worked
by Maurice McHugh, STAR, NDE, OSD, and OSPO personnel. Reformatting Toolkit developers intend to participate in these meetings and discussions.
· Status: Closed. We recommend closing this. The updated SPSRB documentation has defined roles and responsibilities for writing of these documents on the STAR side. At TRR new project-level risk was opened to address NDE needing to modify their contract to complete this delivered documentation in time for transition to OSPO.
36
PDR Risk· Risk 3: There are small variations in the types of platforms and the
versions of the compilers· Assessment: Low· Impact: May obtain different results using different compilers.· Likelihood: Low· Mitigation: Reformatting Toolkit developers will work with NDE during
system tests in the integration and production phase to ensure that those results match the results from the units. The NDE Build Content Reviews and delivery of DAP prototypes should help to resolve these issues early on in development.
· Status: Open
37
PDR Risk
· Risk 4: Data format translation may involve some unit conversion and possible reduction of precision (significant digits)
· Assessment: Low· Impact: Any modification of the data from its original form may not be
apparent to the user.· Likelihood: Low· Mitigation: Document data manipulations in the NDE Delivered
Algorithm Package (in place of ATBD). This will be done in the System Maintenance Manual (SMM).
· Status: Open
38
PDR Risk· Risk 6: Risk on NDVI and snow mask. Have not yet demonstrated
ability to encode GRIB2 files.· Assessment: Moderate· Impact: Failure to meet the requirement to have demonstrated GRIB2
tailoring capability.· Likelihood: Low· Mitigation: NDVI and snow mask will not be produced. However,
GVF will need to be in GRIB2 format, but that’s something that will not be a capability of the toolkit until Phase 1 EDR DAP. This capability will be demonstrated at the Phase 1 EDR & Phase 2 EDR products TRR scheduled for 1/23/2012.
· Status: Open.
39
CDR Risks and Actions
40
CDR Risk· Risk 14: STAR and NDE CM need to collaborate to discuss formats,
defining fields, the process for delivering DAPS with an official naming convention.
· Assessment: Low· Impact: Confusion and delays caused by not having common naming
conventions and not having standard DAP delivery process defined.· Likelihood: Low· Mitigation: STAR and NDE CM need to collaborate to discuss
formats, defining fields, the process for delivering DAPS with an official naming convention.
· Status: Closed. We recommend closing as all this is covered in the NDE document entitled “Algorithm Delivery Standards, Integration, and Test Version 1.3”.
41
CDR Risk· Risk 15: Need to identify an independent code review· Assessment: Low· Impact: Reduction in the quality of the delivered software.· Likelihood: Low· Mitigation: Document plan for an independent code review.
Understand NESDIS QA process is being defined. · Status: Closed. We recommend closing this as this project schedule
has TRR/CTRs for each phase and has added software reviews (code walk-throughs) for all future phases. OSPO and NDE are invited to participate in these reviews and OSPO will be expected to participate in the software reviews. The question is how much effort can groups outside STAR afford to invest given current budgets. If the reviewers agree this can be sufficient, this risk can be closed.
42
CDR Risk· Risk 16: EMC are concerned that the conversion of HDF5 to
netCDF4 may add significant time to the delivery of the products.· Assessment: Low· Impact: Increase in product latency and failure to meet customer
needs.· Likelihood: Low· Mitigation: Need to conduct tests to verify that the conversion times
meet latency requirements.· Status: Closed. We recommend closing this given that the tests
shown here demonstrate that the conversion times were sufficient to maintain latency.
43
CDR Risk· Risk 17: Encourage NWS (EMC, AWIPS) personnel to take part in
future reviews, major meetings, as well as working-level meetings. JCSDA and EMC were lightly represented in the CDR.
· Assessment: Low· Impact: Changing user needs may not be regularly communicated to
the developers.· Likelihood: Low· Mitigation: May need to establish a working group that facilitates this
communication.· Status: Closed. We recommend closing this as regular monthly
EMC/NDE/STAR meetings are held (Kevin Berberich). In addition, EMC & JCSDA representatives are always invited to this project’s reviews. Tom Schott’s VIIRS TIM helped resolve many remaining VIIRS SDR BUFR issues for Phase 1.
44
CDR Risk· Risk 18: ESPC may not have the BUFR expertise to maintain the
toolkit.· Assessment: Low· Impact: OSPO may not be able to maintain the toolkit in the future.· Likelihood: Low· Mitigation: OSPO is getting FY12 funding to support the transition
and maintenance from NDE on the OSPO side.· Status: Open. At TRR it was decided that this is a project-level risk
that should remain open. The N4RT project plan anticipates STAR will provide upgrades and maintenance through 2014.
45
CDR Risk· Risk 19: Should OMPS NP and TC be in the same BUFR file?· Assessment: Low· Impact: Flexibility of FOV for OMPS NP pose a problem for converter
if combined.· Likelihood: Low· Mitigation: These will be in separate files.· Status: Closed. We recommend closing this.
46
Summary· There are 11 risks:
»5 from PDR»6 from CDR
· We recommend closing: »2 of the PDR risks.»5 of the CDR risks.
· 4 risks will remain open.
47
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
48
Section 3 – Requirements
Presented by
Thomas KingRiverside
49
Requirements Overview· SPSRB Requirements were presented to the developers in a document entitled:
“Level 1 Requirements for a NetCDF4 Reformatting Tool” (Version 1.5).
· Product requirements have been added to those from the SPSRB and are presented here as well. These additional requirements were obtained in a series of meetings between the developers, the customers (EMC/JCSDA and EUMETSAT) and the heritage product teams.
· Using all of this information, a Requirements Allocation Document (RAD) has been generated for the Reformatting Toolkit project.
· All current Phase 1 and Phase 2 requirements are listed, but we’ll focus mainly on those for Phase 1 SDR.
· Text highlighted » Yellow indicate basic requirements» Orange indicates new, modified, or removed requirements since CDR.
50
Phase I User-to-STAR Mapping
Prioritized Product EMC User Contacts
STAR/OSPO Product and Heritage Contacts
Heritage Product
ATMS Radiances Dennis Keyser, John Derber
Dennis Keyser, John Derber, Sid Boukabara
AMSU, MHS, HSB
CrIS Radiances Jim Jung, Dennis Keyser
Jack Woollen, Simon Elliott, Tom King
IASI, AIRS
VIIRS Radiances Dennis Keyser, John Derber
Dennis Keyser, John Derber
AVHRR GAC
Nadir Profile and Total Column Ozone (OMPS)
Dennis Keyser Larry Flynn, Donna McNamara
SBUV, GOME
Aerosol Optical Thickness Jeff McQeen Paul Haggerty MODIS Aerosols
Sea Surface Temperature Bert Katz, William Gemmill
Shasha Ignatov, John Sapper, Robert Grumbine
AVHRR derived SST (ACSPO)
White = Phase 1 SDR BUFRGray = Phase 1 EDR BUFR
51
Requirement 1.0· Requirement 1.0: STAR shall deliver to NDE a
reformatting toolkit capable of translating NESDIS NetCDF4 data products into NCEP-accepted data formats (i.e., BUFR and/or GRIB2).
52
Requirement 1.0· Requirement 1.1: The toolkit shall be capable of reformatting the NPP tailoring prioritized
phase 1 product list.
· Requirement 1.2: The toolkit shall provide its capabilities such that it may be run automatically within an operational system, especially within the NDE environment.
· Requirement 1.2.1: The Toolkit shall compile and run on the NDE IBM AIX P5, P6, and P7 series hardware.
· Requirement 1.2.2: The Toolkit shall be able interact with the NDE Data Handling System (DHS).
· Requirement 1.2.2.1: The Toolkit shall be able to read a Production Control File (PCF).
· Requirement 1.2.2.2: The Toolkit shall handle and return errors according to NDE/STAR standard codes.
· Requirement 1.2.2.3: The Toolkit shall be able to write a (Product Status File) PSF.
53
Requirement 1.0· Requirement 1.3: The toolkit shall consist of modular components that can be tested
independently and data should be stored in allocatable structures.
· Requirement 1.3.1: The code shall consist of a single compiled program that parses arguments and logically assigns tasks to a family of hierarchically structured tailoring subroutines.
· Requirement 1.4: STAR shall include one update to the reformatting toolkit within its initial project plan.
· Requirement 1.5: STAR shall propose additional updates to the reformatting toolkit at a future Annual Review for Satellite Product Development that will address the NDE Phase 2 products.
· Requirement 1.6: STAR shall use the standard set of NCEP software libraries for BUFR and GRIB2 in the reformatting toolkit.
54
Requirement 1.0· Requirement 1.7: STAR shall update the reformatting toolkit when NCEP updates its BUFR
and GRIB2 libraries.
· Requirement 1.7.1: Updates shall be made when there are updates to the versions of the NetCDF4 library being used by NDE.
· Requirement 1.8: STAR shall coordinate with the NDE Project before proposing any enhancements to add other standard format translations to the toolkit at the Annual Review for Satellite Product Development.
· Requirement 1.9: The output from the toolkit shall be compared with the input to verify that the conversion was performed correctly.
· Requirement 1.10: The translation toolkit shall convert from the new format back into NetCDF4.
· Requirement 1.11: The reformatting software shall log each transaction’s control information, including: the calling application, the type of transaction requested, the start and end times, and completion status codes.
55
Requirement 1.0· Requirement 1.11.1: The Reformatting Toolkit software shall generate run logs (contain
human-readable messaging and time stamps) and return NDE/STAR standard (agreed upon) error codes to the DHS.
· Requirement 1.12: Applications running under either Linux or AIX Operating Systems shall be able to provide the reformatting toolkit data and be able to accept the data from the toolkit for further processing (e.g., dissemination).
· Requirement 1.13: The toolkit parameters (e.g., how to use the service) shall be well documented.
· Requirement 1.13.1: Reformatting Toolkit Developers shall provide documentation in the form of a tailored Delivered Algorithm Package (DAP) whose name and contents are defined in the NDE document entitled “Algorithm Delivery Standards, Integration, and Test V1.3”.
· Requirement 1.13.2: The DAP shall contain the following two SPSRB documents: the SMM (System Maintenance Manual) and the EUM (External Users Manual) as well as the Test Readiness Document.
· Requirement 14.1: The messages provided by the toolkit in the event of failure to perform a requested service shall be comprehensible by untrained operators.
56
Requirement 1.0· Requirement 1.14.1: Reformatting Toolkit shall use the standard set of error return codes
developed by NDE for code running within the DHS.
· Requirement 1.15: The messages provided by the toolkit in the event of failure to perform a requested service shall include diagnostic details needed for troubleshooting.
· Requirement 1.15.1: All messaging shall be directed to a run log file. These messages shall be documented in the Reformatting Toolkit tailored DAP.
· Requirement 1.16: STAR shall coordinate development of the reformatting toolkit with the NDE contractors and assist the NDE contractors with the integration of the toolkit within each of the environments of the NDE processing system.
· Requirement 1.17: Toolkit code shall adhere to the SPSRB coding standards.
· Requirement 1.18: Performance shall be measured on a product level.
· Requirement 1.19: The Toolkit shall output BUFR and GRIB2 files whose names adhere to the NDE file naming convention in “Algorithm Delivery Standards, Integration, and Test V1.3”.
57
Requirement 2.0· Requirement 2.0: STAR shall provide monthly
project status reports to OSPO and OSD.
58
Requirement 3.0· Requirement 3.0: Earned Value Management shall
be performed on the project.
59
Requirement 4.0· Requirement 4.0: STAR shall update the project plan
on an annual basis and submit it to the Annual Review of Satellite Product Development for funding consideration.
60
Requirement 5.0· Requirement 5.0: The toolkit shall be implemented
and tested six months before the NPP launch to ensure NDE readiness.
61
Requirement 6.0· Requirement 6.0: The Reformatting Toolkit shall
tailor the NUCAPS thinned CrIS Radiances from NetCDF4 into BUFR for EMC.
62
Requirement 6.0· Requirement 6.1: The Reformatting Toolkit developers shall work with EMC
and the rest of the NWP community to create a BUFR table for the NUCAPS thinned and full resolution radiances based on AIRS and IASI.
· Requirement 6.2: The table shall use delayed replication for storing the radiances.
· Requirement 6.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)
· Requirement 6.4: The BUFR format shall allow for the storage of negative radiances.
· Requirement 6.5: The file shall contain the following data fields (see table next slide):
63
Product Requirements:CrIS Radiances
Variable Name Variable Name Variable Name Variable Name
Satellite ID Latitude Height of Land Surface Start Channel (per band)
ID of Originating Center
Longitude Satellite Height End Channel (per band)
Satellite Instrument Satellite Zenith Angle Land Fraction Wavenumber
Satellite Classification Satellite Azimuth Land/Sea Qualifier Calibration Quality Flags
Year Solar Zenith Cloud Cover Field of View Quality Flags
Month Solar Azimuth Height of Cloud Top Geolocation Quality
Day Ascending/Descending flag
Radiance Type Flags NUCAPS Quality
Hour Scan Line Number Scan-Level Quality Flags Channel Radiance
Minute Field of Regard Type of Band In dir. of North Pole, distance from the Earth's center
Second Field of View Starting Wavenumber (per band)
In direction of 0 deg E, distance from Earth's center
Location of Platform Orbit Number Ending Wavenumber (per band)
In direction of 90 deg E, distance from Earth's center
64
Requirement 6.0· Requirement 6.6: The reformatting Toolkit shall use the thinned CrIS
radiances (399 channels) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EMC.
· Requirement 6.7: The reformatting Toolkit shall use the full spatial and spectral resolution CrIS radiances (1305 channels and all FOVs on all FORs) files from NUCAPS as an input for generating the CrIS radiance BUFR files for EUMETSAT.
65
Requirement 7.0· Requirement 7.0: The Reformatting Toolkit shall tailor
the JPSS ATMS SDR and TDR data from NetCDF4 into BUFR for EMC.
66
Requirement 7.0· Requirement 7.1: The ATMS BUFR file shall contain, from all channels, the
antenna and brightness temperatures, associated Quality Flags, and the geolocation data at native resolution (not resampled) data.
· Requirement .72: The Reformatting Toolkit developers shall work with EMC and the MIRS team to create an ATMS BUFR table. The ATMS BUFR file shall be based on what is currently provided for AMSU and MHS.
· Requirement 7.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)
· Requirement 7.4: The file shall contain the following data fields (see table next slide):
67
Product Requirements:
ATMS RadiancesVariable Name Variable Name Variable Name
Satellite ID FOV Number Solar Azimuth
ID of Originating Center Granule-Level Quality Flags ATMS Channel Number
ID of Originating Sub-Center Scan-Level Quality Flags ATMS Central Frequencies
Satellite Instrument Geolocation Quality ATMS Channel Bandwidth
Satellite Classification Channel-Level Quality Flags Antenna Polarization
Year Orbit Number Antenna Temperatures
Month Latitude Brightness Temperatures
Day Longitude NeDT cold target
Hour Satellite Height NeDT warm target
Minute Satellite Zenith Angle Satellite Antenna Correction Version Number
Second Satellite Azimuth
Scan Line Number Solar Zenith
68
Requirement 7.0· Requirement 7.5: The Reformatting Toolkit shall use the JPSS ATMS TDR
and SDR files and associated Geolocation files tailored into NetCDF4 as an input for generating the ATMS radiance BUFR files.
69
Requirement 8.0· Requirement 8.0: The Reformatting Toolkit shall tailor
JPSS OMPS Ozone products from NetCDF4 into BUFR for EMC.
70
Requirement 8.0· Requirement 8.1: The product shall contain OMPS Nadir Profile and Total
Column (multiple triplet algorithm, not v8, but output is equivalent to v8 according to Larry Flynn) in separate files.
· Requirement 8.2: The Reformatting Toolkit developers shall work with EMC to develop an OMPS BUFR table based on that currently used for GOME and SBUV.
· Requirement 8.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)
· Requirement 8.4: This file’s content is currently under development.
71
Product Requirements:
OMPS Nadir ProfileVariable Name Variable Name Variable Name
Satellite ID Satellite Zenith Angle Significance and Volumetric Mixing Ratio
ID of Originating Center Satellite Azimuth SO2 Index
ID of Originating Sub-Center Solar Zenith Volcano Contamination Index
Satellite Instrument Solar Azimuth SBUV Total Ozone Quality
Year Satellite Height SBUV Profile Ozone Quality
Month Orbit Number Geolocation Quality
Day Asc/Desc Flag Ozone Quality Flag
Hour Vertical Significance
Minute Pressure
Second Number of Retrieved Layers
Latitude Total Ozone
Longitude Ozone p (Dobson Units)
72
Requirement 8.0· Requirement 8.5: The Reformatting Toolkit shall use the JPSS OMPS Total
Column Ozone EDR files and OMPS Nadir Profile IP files tailored into NetCDF4 as an input for generating the OMPS Ozone BUFR files.
73
Requirement 9.0· Requirement 9.0: The Reformatting Toolkit shall tailor
JPSS VIIRS SST products from NetCDF4 into BUFR for EMC.
74
Requirement 9.0· Requirement 9.1: Product shall contain Skin SST, Bulk SST, Quality Flags, and
geolocation data.
· Requirement 9.2: Reformatting Toolkit developers shall work with EMC to create a BUFR table for the VIIRS SST product.
· Requirement 9.3: The VIIRS SST BUFR table shall be derived from that currently being used for the AVHRR derived SST (from ACSPO - Advanced Clear-Sky Processor for Oceans).
· Requirement 9.4: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)
· Requirement 9.5: The file shall contain the following data fields (see table next slide):
75
Product Requirements:
VIIRS SSTVariable Name Variable Name Variable Name
Satellite ID Latitude Pixel type
ID of Originating Center Longitude Asc/Desc flag
ID of Originating Sub-Center Satellite Zenith Angle Geolocation Quality
Satellite Instrument Satellite Azimuth VIIRS Geolocation Quality
Satellite Classification Solar Zenith Retrieval Data Quality
Year Solar Azimuth Adjacency Cloud Mask
Month Satellite Height SST Pixel-Level Quality flag
Day Scan Line Number SST
Hour FOV number SST bulk
Minute Orbit Number
Second Day/Night flag
76
Requirement 9.0· Requirement 9.6: The Reformatting Toolkit shall use the JPSS VIIRS SST
EDR files and associated geolocation files tailored into NetCDF4 as an input for generating the VIIRS radiance BUFR files.
77
Requirement 10.0· Requirement 10.0: The Reformatting Toolkit shall
tailor JPSS VIIRS radiances, brightness temperatures, and reflectances from NetCDF4 into BUFR for EMC.
78
Requirement 10.0· Requirement 10.1: Each BUFR file will contain the VIIRS data for a single
band (Imagery band, Moderate band, or Day/Night band resolution).
· Requirement 10.2: Coverage will be global.
· Requirement 10.3: Each of the 3 bands will use the same VIIRS BUFR table.
· Requirement 10.4: Removed: The product shall contain the land and cloud mask if it doesn’t take too long for the IDPS to generate those EDRs.
· Requirement 10.5: Reformatting Toolkit developers shall work with EMC and the rest of the NWP user community to create a VIIRS BUFR table derived from that used earlier for the GAC AVHRR.
79
Requirement 10.0· Requirement 10.6: BUFR messages shall be smaller than 50KB.
(Removed: this is no longer a limitation of the BUFRLIB)
· Requirement 10.7: The file shall contain the following data fields (see table next slide):
80
Product Requirements:
VIIRS RadiancesVariable Name Variable Name Variable Name
Satellite ID Latitude VIIRS Geolocation Quality
ID of Originating Center Longitude Radiance Quality
ID of Originating Sub-Center Satellite Zenith Angle Type of Band
Satellite Instrument Satellite Azimuth Cloud Mask
Satellite Classification Solar Zenith Channel Number
Year Solar Azimuth Channel Wavelength
Month Satellite Height Channel Radiance
Day Orbit number Channel Reflectance
Hour Scan line number Surface Type
Minute FOV number
Second Geolocation Quality
81
Requirement 10.0· Requirement 10.8: The BUFR shall contain the following Moderate and
Imagery resolution channels as requested by EMC:– Channel M12 (3.70 microns)– Channel M13 (4.05 microns)– Channel M15 (10.763 microns)– Channel M16 (12.013 microns)– Channel I5 (11.450 microns)
82
Requirement 10.0· Requirement 10.9: The toolkit shall not write data for obviously cloudy VIIRS FOVs.
This will reduce the total data volumes. Obviously cloudy FOVs are those with brightness temperatures in one of the longwave SST channels (M15 or M16) less than 270K. To preserve sea ice FOVs, this test shall only be applied for latitudes equatorward of 50 degrees. (Requirement removed – see below)
Note: An important concern is that more CPU time required to encode the BUFR than what NDE can supply. A recommended solution was to reduce the number of VIIRS channels to only a required subset, reject obviously cloudy data, and reject land-only granules. Very few granules, if any, will be entirely land-only. This would also require comparing the lat/lons of all points to a land mask which means more processing and we'd still have to read all the data anyway.
Andrew Collard in an email on 6/8/2011 says "We don't NEED data thinning, we can ACCEPT data thinning." Therefore, we have decided to supply all the granules, but reduce the channels and employ the obviously-cloudy check.
Removed, Bob Grumbine wants to keep frozen inland lakes, to do this we decided to not thin. This is documented in an email from Andrew on 11/9/2011. CC’d were Bob Grumbine, John Derber, Tom Schott, and Walter Wolf.
83
Requirement 11.0· Requirement: The Reformatting Toolkit shall tailor
JPSS Aerosol Optical Thickness (AOT) from NetCDF4 into BUFR for EMC.
84
Requirement 11.0· Requirement 11.1: The product shall contain the AOT, wavelength of AOT,
and Aerosol Size.
· Requirement 11.2: Reformatting Toolkit developers shall work with EMC to develop the AOT BUFR table based on what has already been done for MODIS.
· Requirement 11.3: BUFR messages shall be smaller than 50KB. (Removed: this is no longer a limitation of the BUFRLIB)
· Requirement 11.4: The file shall contain the following data fields (see table next slide):
85
Product Requirements:VIIRS Aerosol Optical
ThicknessVariable Name Variable Name Variable Name
Satellite ID Latitude Asc/Desc Flag
ID of Originating Center Longitude Geolocation Quality
ID of Originating Sub-Center Satellite Zenith Angle VIIRS Geolocation Quality
Satellite Instrument Satellite Azimuth Retrieval Quality
Satellite Classification Solar Zenith Aerosol Type (land)
Year Solar Azimuth AOT Quality Flag
Month Satellite Height Aerosol Wavelength Angstrom Exponent
Day Orbit Number Channel Wavelength
Hour Scan Line Number Optical Depth
Minute FOV Number
Second Remotely Sensed Surface Type
86
Requirement 11.0· Requirement 11.5: The Reformatting Toolkit shall use the JPSS Aerosol
Optical Thickness EDR files and associated Geolocation files tailored into netCDF4 as an input for generating the Aerosol Optical Thickness EDR BUFR files.
87
Requirement 12.0· Requirement 12.0: The Reformatting Toolkit shall
tailor NDE-generated Polar Winds product from NetCDF4 to BUFR format for EMC.
· Additional requirements for this product will be forthcoming.
88
Requirement 13.0· Requirement 13.0: The Reformatting Toolkit shall
tailor NDE-generated Green Vegetation Fraction products from NetCDF4 to GRIB2 format for EMC.
· Additional requirements for this product will be forthcoming.
89
Interface Requirements· Requirement 14.0: The Reformatting toolkit shall
comply with OSPO coding standards identified in the OSPO security checklist.
90
Requirements Summary
· The Reformatting Toolkit developers have met with NDE and all the users of the original Phase I NDE tailored products.
· All Reformatting Toolkit project, functional, and product requirements have been captured and documented in the Reformatting Toolkit RAD.
· As development continues, the detailed product requirements shall continue to be refined.
91
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
92
Section 4 –Quality Assurance
Presented by
Thomas KingRiverside
93
Quality AssuranceBackground
· STAR is using the Capability Maturity Model Integrated (CMMI) to improve processes and practices for development and the transfer of research to operations.
· The STAR Enterprise Life Cycle (EPL) merges the CMMI and SPSRB standards.
· This development group has experience with IASI and NUCAPS. They are both pathfinder projects (CMMI Level 3). The IASI system has been transitioned to operations successfully.
· The Reformatting Toolkit algorithm development is following the same process but tailored to the scale of the project.
94
· The Reformatting Toolkit Preliminary Design Review (April 2009) » Present the initial draft of the requirements and discuss a proposed design.» An Reformatting Toolkit Requirements Allocation Document (RAD) has been made
available to Reformatting Toolkit stakeholders. It will be updated throughout the lifecycle of the project.
· The Reformatting Toolkit Critical Design Review (September 2009)» To finalize requirements and to verify that the chosen design is able to meet those
requirements.
· An Reformatting Toolkit Test Readiness Review/Code Test Review (Oct 2011) » Present the unit test plans and test results to demonstrate that the toolkit is ready to
be run in the Test Environment.
· The Reformatting Toolkit Algorithm Readiness Review (Nov 2011) » Demonstrate that the code has been system tested at STAR, software has met
coding standards, and that all documentation is complete and ready for the official Phase 1 SDR DAP delivery.
Quality Assurance – Phase 1 SDR
Implemented Tailored STAR EPL Reviews
95
· CM Tool (IBM Rational ClearCase, Version 7.0 ) » Has been purchased and implemented in the Collaborative
Environment.
· CM personnel have been identified.
· CM training:» Administrator training completed. » Developers will be trained by the CM administrator.
· We are using a modified version of the CM plan developed for GOES-R framework being developed at STAR.
· The Reformatting Toolkit software has been checked into ClearCase.
Configuration Management (CM)
96
Coding Standards · We follow the SPSRB-approved coding standards developed by
the Common Standards Working Group (composed of STAR, OSPO, NDE personnel).
· Coding standards guidelines and quick references are available.
· Provide a common list of abbreviations.
· Adhere to the standards throughout the development life cycle.
· Have checklists available for developers to keep track of the delivery status of the code.
97
Quality Assurance – Software
· All code development is being conducted on a platform that is nearly identical to the NDE integration and production target platforms using the same compilers and operating system.
· Only the latest and official releases of the NCEP BUFRLIB, GRIB2, and NetCDF4/HDF5 libraries are used in the software and these match to what is available on NDE’s SADIE platform.
· STAR code checking tools are used to minimize coding bugs and to ensure that Reformatting Toolkit code meets STAR coding standards.
· Unit tests are conducted on each software unit to demonstrate that products can be correctly produced. Test methods and results are documented and presented to stakeholders.
98
Quality Assurance – Software
· Two prototype versions of the Toolkit have be tested in the NDE hardware. The final DAP will be delivered at the end of November 2011.
· Customers have access to test BUFR/GRIB2 data products to verify required content is present and to test ingest.» From STAR simulated CrIS/ATMS datasets» From JPSS P72 datasets
· Software Code Reviews will be conducted on future phases of this project. » Comply with SPSRB coding standards» Comply with OSPO security checklist
99
Quality Assurance – Products
· Reformatting Toolkit developers will work with EMC, NRL, FNMOC, GMAO, EUMETSAT, and UK Met users to ensure that the proper BUFR descriptors are used in the BUFR and GRIB2 tables and that, whenever possible, they are WMO approved (as opposed to local). » Chosen descriptors must work with NCEP BUFRLIB.
· Reformatting Toolkit developers will work with EMC, heritage product developers, and JPSS operational algorithm teams to ensure consistency with heritage products with respect to format and content.
· Reformatting Toolkit developers will make BUFR tables and test products (BUFR and GRIB2) available to users before the operational products are made available. This will allow for preliminary product content validation.» CrIS and ATMS are currently being simulated at STAR.» For the other products such as VIIRS, we use NDE P72 data.
100
Current Status of BUFR Table
Development
File Table Status Test Data AvailableCrIS SDR BUFR Table
Finished and approved by WMO Yes (July 2009)
ATMS SDR BUFR Table
Finished and approved by WMO Yes (October 2009)
VIIRS SDR BUFR Table
Finished and approved by WMO Yes (June 2010)
OMPS EDR BUFR Table
OMPS Nadir Profile table has been finished and is going through the WMO approval process.
OMPS Total Column table is being developed.
No
AOT EDR BUFR Table
Finished and going through the WMO approval process.
No
SST EDR BUFR Table
Finished and going through the WMO approval process.
No
101
Phase 1 Sample Product Availability
· First CrIS and ATMS BUFR sample data were made available to NCEP - 2/20/2009
· CrIS ~300-channel and ATMS BUFR made continuously available on STAR server - 10/9/2009
· First VIIRS sample BUFR files were made available to NCEP - 1/8/2011
· First Prototype N4RT DAP - 5/25/2011
· CrIS 1305-channel were made available - 5/31/2011
· VIIRS BUFR updated to only 4 Moderate and 1 Imagery band - 7/28/11
· CrIS BUFR updated to final 399-channel set - 8/3/11
· Second Prototype N4RT DAP (in support of NCO testing) - 9/2/2011
102
Quality Assurance – Archive and Maintenance
· Archive Plan» Reformatting toolkit will be integrated into NDE system. Our understanding
is that NDE will archive the N4RT DAP with the other NDE DAPs.» Product archive requirements are already addressed within product
development projects.– JPSS program office works with CLASS to archive xDRs– NOAA Unique Product (NUP) projects work with CLASS as required– There are no plans to archive N4RT output products since these can be
regenerated from the SDR, TDR, and EDR.
· Long Term Maintenance Plan» The reformatting toolkit will be maintained by the OSPO staff» STAR system developers will be available
103
Quality Assurance – Documentation
· The following documents will be created and included as part of the official DAP delivery
– SMM (System Maintenance Manual)– EUM (External User’s Manual)– No ATBD– TRD (Test Readiness Document)
· These additional STAR EPL documents will not be delivered, but will be available to reviewers on the project website:
– RAD (Requirements Allocation Document)– CDD (Critical Design Document)– PDD (Preliminary Design Document)– Toolkit RID (Review Item Disposition)– No SA (Submission Agreement) with archive
104
Quality AssuranceSummary
· Quality assurance plan will consist of: » Project reviews at which stakeholders are encouraged to participate.» Ongoing interaction with customers, heritage product developers,
operations, NDE, and the SPSRB.» Adhering to STAR/NDE software standards and use of standard
libraries only.» Software unit tests shall be presented in the TRR/CTR.» Documentation of the Reformatting Toolkit code operation, production
rules, and software tests will be in the DAP.» Documentation of requirements will be in the Reformatting Toolkit
RAD.» Early release of BUFR and GRIB2 products, tables, and additional
resources will allow for customer validation of products.
105
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software Architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
106
Section 5 –Software
Architecture
Presented by
Thomas KingRiverside
107
Reformatting Toolkit Architecture
· Hardware Environment» Development and Unit testing» Test Product Distribution» System Testing and Integration» Production
· Data Files» Input Files» Static/Ancillary Files» Output Files» Run Files
· Software Description» External Interfaces» System Level» Unit Level
108
Reformatting Toolkit Development Hardware
· Reformatting Toolkit Development Hardware - Unit tests were conducted on the orbit002b development machine at STAR. This machine is maintained by STAR IT.» IBM P6 (AIX 6.1)» 9.6 TB disk space» 14 CPU (4208 MHz)» 46366 MB Memory» IBM XL Version 10.1 C/C++» IBM XL Version 12.1 Fortran 77/90
· Configuration Management of Reformatting Toolkit code is being conducted in the STAR Collaborative Environment on STAR Linux hardware running IBM Clear Case.
109
Reformatting Toolkit Test Product Distribution
· STAR Data Server - Reformatting Toolkit test products are available on a distribution server at STAR (ftp2.orbit.nesdis.noaa.gov). » Linux» 3.2 TB disk space» Access via anonymous ftp» Products available on a near real time basis
· This server hosts:» CrIS SDR BUFR – From simulation continuously available 399 and 1305 channels at full
spatial resolution» ATMS SDR/TDR BUFR – From simulation continuously available at full spatial resolution» VIIRS SDR BUFR – 1 day from the P72 data set for 4 M-band and 1 I-band channels
· The BUFR tables of OMPS Nadir Profile, VIIRS EDR SST, and VIIRS EDR AOT are currently going through the WMO approval process.» When sample data are ready for these products, they will also be made available through
the ftp2 server.
110
Reformatting Toolkit Integration and
Production Hardware · NDE SADIE – The Reformatting Toolkit system testing and
integration will be conducted on the SADIE integration platform working with NDE integration personnel. This hardware is located at NSOF and is maintained under the ESPC maintenance contract. » IBM P6 (AIX 6.1)» 52 TB disk space» 8 CPUs (4204 MHz)» 63232 MB Memory» IBM XL Version 11.1 C/C++» IBM XL Version 13.1 Fortran 77/90
111
Reformatting Toolkit Integration and
Production Hardware · NDE Test/Production Hardware - After successful system
tests, NDE will check the code into their CM and then promote it to the Test/Production machine. This machine will be located at NSOF. As per email with Peter MacHarrie these will be the specs at launch:» IBM P6 (AIX 6.1)» 96 AIX CPUs» 48 Intel CPUs» No compilers installed
112
Reformatting Toolkit Data
· The tables on the following slides show all the Reformatting Toolkit input and output.
· The tables contain all files for Phase 1 and Phase 2 tailoring. » Phase 1 SDR related files are in light blue shaded cells.» Phase 1 EDR and Phase 2 EDR related files are in gray shaded cells.» Phase 3 information is not yet shown as the requirements are still
being developed for these products.
· Ancillary/Static files such as the BUFR tables are listed.
· All Phase 1 SDR output will be in BUFR.
113
Reformatting Toolkit Input Data Files
File Format Source Description PurposeCrIS thinned SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned 399 channel radiances
for all FOVs.CrIS BUFR
CrIS all-channel SDR Radiances NetCDF4 NUCAPS The NUCAPS CrIS thinned 1305 channel radiances for all FOVs.
CrIS BUFR
ATMS TDR NetCDF4 JPSS JPSS ATMS full resolution file tailored into NetCDF4 from the JPSS HDF5.
ATMS BUFR
ATMS SDR NetCDF4 JPSS JPSS ATMS full resolution file tailored into NetCDF4 from the JPSS HDF5.
ATMS BUFR
ATMS SDR/TDR Geo NetCDF4 JPSS JPSS ATMS Geolocation file that is associated with the ATMS SDR and TDR. This is the same file.
ATMS BUFR
VIIRS M-Band 12 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 12.
VIIRS BUFR
VIIRS M-Band 13 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 13.
VIIRS BUFR
VIIRS M-Band 15 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 15.
VIIRS BUFR
VIIRS M-Band 16 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 16.
VIIRS BUFR
VIIRS I-Band 5 SDR NetCDF4 JPSS JPSS VIIRS Moderate resolution band SDR for channel 5.
VIIRS BUFR
114
Reformatting Toolkit Input Data Files
File Format Source Description PurposeVIIRS M-Band SDR Geo NetCDF4 JPSS VIIRS Moderate resolution band SDR Geo file. VIIRS BUFR
VIIRS I-Band SDR Geo NetCDF4 JPSS VIIRS Imagery resolution band SDR Geo file. VIIRS BUFR
OMPS Nadir IP NetCDF4 JPSS JPSS OMPS Nadir Profile ozone Intermediate Product file tailored into NetCDF4. Geolocation information is currently in the file. NGAS documentation indicates that much of the content of this product is TBD.
OMPS Nadir Profile BUFR
OMPS Total Column EDR NetCDF4 JPSS JPSS OMPS Total Column ozone EDR file tailored into NetCDF4.
OMPS TC BUFR
OMPS Total Column SDR Geo NetCDF4 JPSS JPSS OMPS Total Column ozone SDR Geo file tailored into NetCDF4. This same Geo file is the used with the OMPS Total Column EDR.
OMPS TC BUFR
115
Reformatting Toolkit Input Data Files
File Format Source Description PurposeAerosol Optical Thickness EDR NetCDF4 JPSS JPSS AOT EDR file tailored into NetCDF4. AOT BUFR
Aerosol Optical Thickness EDR Geo NetCDF4 JPSS JPSS AOT EDR Geo file tailored into NetCDF4. AOT BUFR
Sea Surface Temperature EDR NetCDF4 JPSS JPSS SST EDR file tailored into NetCDF4. SST BUFR
Sea Surface Temperature EDR Geo NetCDF4 JPSS JPSS Moderate Resolution Terrain-Corrected Geolocation file tailored into NetCDF4. This is to be used with the JPSS SST EDR.
SST BUFR
Polar Winds EDR NetCDF4 Polar Winds Product Processing
The Polar winds EDR produced by Jeff Key and Jamie Daniel’s product system running within NDE
BUFR
GVF EDR NetCDF4 GVF Product Processing
The Green Vegetation Fraction EDR produced by Ivan Csiszar’s product system running within NDE.
GRIB2
116
Reformatting Toolkit Ancillary/Static Data
Files
File Format Source Description Purpose
CrIS SDR BUFR Table ASCII N4RT CrIS radiances (399 and 1305 channels, all FOVs) BUFR Template
ATMS SDR BUFR Table ASCII N4RT ATMS radiances full resolution BUFR Template
VIIRS SDR BUFR Table ASCII N4RT VIIRS radiances full resolution BUFR Template
OMPS EDR BUFR Table ASCII N4RT OMPS Nadir Profile and Total Column Ozone BUFR Template
AOT EDR BUFR Table ASCII N4RT Aerosol Optical Thickness and Particle Size BUFR Template
SST EDR BUFR Table ASCII N4RT Sea Surface Temperatures, Cloud Mask BUFR Template
PW BUFR Table ASCII N4RT Polar Wind products EDR BUFR Template
GVF GRIB2 Table ASCII N4RT Green Vegetation Fraction EDR GRIB2 Template
117
Reformatting Toolkit Output Data Files
File Format Description Users
CrIS SDR BUFR CrIS thinned radiances (399 channels, all FOVs) JCSDA, NCEP
CrIS SDR BUFR CrIS thinned radiances (1305 channels, all FOVs) EUMETSAT, UKMet, ECMWF
ATMS SDR BUFR ATMS radiances full resolution JCSDA, NCEP
VIIRS M-band SDR BUFR VIIRS radiances moderate band JCSDA, NCEP
VIIRS I-band SDR BUFR VIIRS radiances image band JCSDA, NCEP
OMPS Nadir EDR BUFR OMPS Nadir Profile Ozone JCSDA, NCEP
OMPS TC EDR BUFR OMPS Total Column Ozone JCSDA, NCEP
AOT EDR BUFR Aerosol Optical Thickness and Particle Size JCSDA, NCEP
SST EDR BUFR Sea Surface Temperatures, Cloud Mask JCSDA, NCEP
PW EDR BUFR Polar Winds JCSDA, NCEP
GVF EDR GRIB2 Green Vegetation Fraction JCSDA, NCEP
118
Reformatting Toolkit Resource and Log
FilesFile Format Source Description PurposePCF ASCII NUCAPS The PCF file containing all the required input
parameters for the N4RT driver script.Driver input
N4RT Run Log ASCII N4RT The N4RT driver script log file containing all the high-level (e.g. file management) standard error and standard output from a given run.
Diagnostic
Executable Log ASCII N4RT The executables log file for a given run. The contains the low-level (e.g. file IO, system-level) error messages.
Diagnostic
N4RT Main Resource ASCII N4RT The internal control file required by the N4RT executable main program.
Executable input
PSF ASCII N4RT The Process Status File containing only the successfully generated output files.
Driver output list
119
Reformatting Toolkit Software: External Interfaces
· The Reformatting Toolkit system will be run via the execution of a single driver script that will be invoked, monitored, and managed by the NDE DHS Product Generation Manager.
· Execution of the script will be event-driven (i.e. the arrival of a required input file whose type is predefined in the Reformatting Toolkit production rules given to NDE).
· An invocation of Reformatting Toolkit by the NDE DHS will be used to perform a single file-to-file (e.g. granule-to-granule) translation (as opposed to reformatting many files as once).
· The driver script is designed to run in a single working directory. All Reformatting Toolkit output will be produced in this same working directory.
· The driver script will require an input PCF file containing parameters needed for file conversions.
120
Reformatting Toolkit Software: External Interfaces
· The driver script will:» Parse the PCF content» Run h5augjpss if necessary» Generate the intermediate resource file for the conversion executable» Run the compiled conversion code» Direct program standard output and error to log files» Generate the return code for the DHS» Generate a PSF
· If there are errors for a given run, NDE will direct the contents of the run in a forensics repository.
· NDE will transfer the output files to the NDE distribution server.
121
Reformatting Toolkit External
Interfaces to NDE
Reformatting Toolkit Driver
Script
SAN
NDE Product Generation Manager
Reformatting Toolkit External Interfaces
Product Generation
Specifications
Working Directory
Systems Configuration
s
Forensics Repository
Input Files &
PCF
InvocationProcess Req.
Rule SetsOutput Files &
PSF
BUFR & GRIB2
Output Files
PSF (N4RT output)
Return Code
Working Directory Output
Input Files (NetCDF4)
NDE DDS
PCF (N4RT input)DAP Specifications
Data AreasConfigurations InfoN4RT SystemNDE Production Manager
NDE DHS Boundary
Input Files (NetCDF4)
122
Reformatting Toolkit Software: System Level
· The Reformatting Toolkit driver script is a single Perl script that acts as a wrapper for the compiled Reformatting Toolkit code.» There are no hard coded paths in the script. All needed information regarding
locations of files will come through the PCF.» All Perl library function calls and system calls have their return values checked
and errors trapped so the exits are graceful and informative.» All standard out and standard error is directed to a single log file that can be
read by NDE for obtaining any error or warning messages.» The driver script translates the low-level program errors into a high-level
numerical value expected by the PGM (0 = success, 1=failure).» The driver will run the h5augjpss conversion utility if the input data type
requires conversion. » The driver script generates an internal control file for the main Reformatting
Toolkit program.
123
Reformatting Toolkit Software: System Level
· The PCF content:» Job coverage start date and time string (yyyymmddhhmmsss)» Job coverage end data and time string (yyyymmddhhmmsss)» Input NetCDF4 file names» Direction of conversion mnemonic (eg. NC2BUFR)» Input and output product types mnemonic (e.g. NUCAPS)» Input BUFR or GRIB2 table needed for conversion» Path to the executables (main_npr and h5augjpss)
· The PSF content (test mode): » Output BUFR or GRIB2 file name
124
Reformatting Toolkit Software: System Level
· The h5augjpss conversion utility» Command line tool run within the driver script upstream of the N4RT conversion program.» The JPSS HDF5 files are not compliant with the netCDF4 data model so they cannot be
read by the netCDF4 library API functions.» h5augjpss is a tool that modifies a JPSS HDF5 file by adding associated data or
metadata or by hiding HDF5 elements in order to make the file accessible to netCDF based applications and tools.
» This is required for the converting ATMS and VIIRS HDF5 files, but not NUCAPS because those files are already processed into netCDF4.
» The HDF Group was funded by JPSS to develop the tool.» Yi Song was one of the early testers of this tool working with the HDF Group to increase
is functionality.
· The following flow chart shows the system-level flow (within the driver script). Black lines identify operational flow and blue show the test flow capabilities that are used for validation.
125
Reformatting Toolkit System
Level Data FlowReformatting Toolkit System Level Data Flow
PCF Return Value to PGM
Execution from PGM
N4RT Driver Script
Working directory
PSF
Working directory
N4RT MainConverter N4RT log
BUFR file
Resource
GRIB2 file
Working directory
NetCDF4
NC Template
BF or GB table
NetCDF4
BUFR/GRIB2
Working directoryNominal Mode
Test Mode
h5augjpss
126
Reformatting Toolkit Software: Unit Level
· The Reformatting Toolkit consists of a single main Fortran 90 program. It contains all the predefined data structures for the various BUFR data types.» APIs for BUFR are only Fortran 77.» Much of the code being reused is already in Fortran 90.» Many users tend to request readers in Fortran 90.
· In the Reformatting Toolkit main program, all data structures are declared and a series of cases statements directs the program flow based on the type of input files and the direction of conversion.
· The Reformatting Toolkit subroutines at the next level down manage:» Allocation, initialization, deallocation of data structures. » They are designed for overseeing specific product definitions and conversions.
127
Reformatting Toolkit Software: Unit Level
· The lowest-level Reformatting Toolkit subroutines:» Perform the actual reading and writing of the specific file types.» Directly call the library APIs for BUFR, GRIB2, and NetCDF4.
· General Reformatting Toolkit compiled code characteristics:» The status of all functions are checked to allow for graceful and informative exits.» No paths are hard coded in the compiled code.» There are no system calls from within the compiled code.» All code is compiled as 64 bit to utilize the IBM architecture.» All code is “mostly statically” linked (i.e. only system libraries like libc.a and the 64-bit
UNIX kernel will be dynamically linked).
· The following flow chart shows the system-level flow (within the driver script). Black lines identify operational flow and blue show the test flow capabilities that are used for validation.
128
Reformatting Toolkit Unit
Level Data FlowReformatting Toolkit Unit-Level Data Flow
Prod N - NC2BF Prod N - BF2NCProd N - NC2GB Prod N - GB2NC
Allocate
Deallocate
Initialize
Prod N - Read BF
Prod N – Write NC
Allocate
Deallocate
Initialize
Prod N - Read NC
Prod N - Write GB
Allocate
Deallocate
Initialize
Prod N - Read GB
Prod N - Write NC
Allocate
Deallocate
Initialize
Prod N - Read NC
Prod N - Write BF
N4RT resource N4RT log file
BUFR GRIB2 NetCDF4 NetCDF4
N4RT Main
Nominal ModeTest Mode
129
Reformatting Toolkit Software: Libraries
· NCEP BUFR Library (available on the NCEP website)» Fortran 77 and a little C code» Compiled as 64 bit» Source code supplied within the N4RT DAP
· NCEP GRIB2 Encoder/Decoder Library (available on the NCEP website)» C and Fortran 90 code» Compiled as 64 bit» Source code supplied within the N4RT DAP
· NetCDF4 Library version 4.1.3 (available from Unidata website)» C code» Compiled as 64 bit» Requires HDF5 1.8.6 and zlib-1.2.5 (available on HDF Group and zlib websites)» Libraries supplied by NDE
· h5augjpss 1.0.0 HDF5 conversion tool (available from the HDF Group)» C code» Supported (Funded) by the JPSS Program Office» Compiled as 64-bit» Source code supplied within the N4RT DAP
130
Toolkit Error Handling· Error checking at all levels.
» Checking the status of all return values from executable program language statements (READ, WRITE, ALLOCATE, DEALLOCATE), script-level functions (Perl), and system calls (from ksh).
» Range checking of values and checks on the sizes of passed arrays are made.
· All errors or noteworthy conditions are trapped. » Script-level log file (NPR.pl.log) will contain the string “Error” or “Warning”. These
strings are identified in the N4RT System Maintenance Manual as strings that NDE can search for production monitoring. If found, the program-level log files can be checked manually for greater detail on the nature of the error.
» Script-level messaging is designed to comply with section 5.1 “Log File Specification” of the NDE document Standards for Algorithm Delivery and Integration Using Delivered Algorithm Packages V1.3.
» Program-level log files are the output logs from the executable code (main_npr.log). This will contain messages labeled as either FATAL, WARNING, or NOTICE are directed to log files.
» Script-level and program-level messages are human-readable explaining the error, location (in program flow or on which file), consequence, and possible cause.
» Logic surrounding the error trapping takes corrective action or exits gracefully.» The return code of the driver script will indicate whether an error has occurred in the
script or down in the executable code.
131
Software Architecture Summary
· Reformatting Toolkit software development, testing, and operation will be conducted on comparable IBM hardware at NSOF and STAR.
· All input and output files have been identified.
· The Toolkit software architecture and program flow has been described.
· The code will use only the official releases or libraries and code will be compatible with those libraries supplied by NDE on SADIE.
· Error handling will be present and messaging will be available to NDE production monitoring.
132
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
133
Section 6 –Unit Tests
Presented by
Yi SongRiverside
134
NC2BF Test Environment
· The development and tests are performed on orbit002b.orbit2.nesdis.noaa.gov – IBM P6.
· This machine is located at STAR and is maintained by STAR IT.» 14 CPUs» 4.208 GHz» 46.336 GB memory» 9.6 TB disk space» AIX 6.1» IBM XL Fortran for AIX, V12.1» IBM XL C/C++ for AIX, V10.1» Perl 5.8.8
· The NetCDF, HDF5 and NCEP BUFR libraries are the latest versions:» hdf5-1.8.7 » netcdf-4.1.3 xlf version 11.1 or greater (Fortran77/90/95)» NCEP BUFRLIB Version 10.0.0
135
NC2BF Test Data Sets· Except CrIS, all the data sets come from the NDE P72 data.
· ATMS data set» ATMS SDR Geolocation » ATMS Science SDR» ATMS TDR
· VIIRS Moderate Bands data set» VIIRS Moderate Bands SDR Geolocation » VIIRS Moderate Resolution SDR Band 12, Band 13, Band 15 and Band 16
· VIIRS Image Bands data set» VIIRS Image Bands SDR Geolocation» VIIRS Imagery band 5 SDR
136
NC2BF Test Data Sets· The CrIS dataset comes from NESDIS/STAR CrIS simulation
observation system that emulates the instrumental and orbital characteristics of the CrIS instruments on the NPP platform to produce:» 2.9 million CrIS spectra/day
· The output of the simulation system is in the CrIS HDF5 JPSS format.
· One program in NUCAPS converts CrIS SDR HDF5 file into two NetCDF4 files» All CrIS channels» One subset with 399 channels.
· Both Geolocation and Science data are included in one CrIS SDR NetCDF4 file
137
NC2BF System Needs
· BUFR table of each product.
· Product type: NUCAP, or ATMS, or VIIRS
· Conversion type: NC2BUFR, or BUFR2NC.
· Channel type: All channels, or subset of channels.
· Input data: Geolocation data and Science data files.
138
System Description – Function and Purpose
· It reads a PCF, writes a resource file and a PSF, and returns error handling information to the calling NDE Data Handling System.
· It converts HDF5 files into NetCDF4 files with utility h5augjpss.
· It reads the resource file into the NC2BF processing chain.
· Decides the converting action: NC2BUFR, or BUFR2NC.
· Applies the BUFR table to the products, and converts them between NetCDF4 and BUFR format.
139
System Test ItemsSoftware Unit Purpose File Input File Output
NPRpl This is a driver script that: (1) Reads in the PCF and parses its contents. (2) Performs local file Management (converts input data file from HDF5 format to NetCDF4 file with utility h5augjpss), generates internal resources file, handles the system calls, and drives main program main_npr. (3) Writes out log files and the PSF file, and returns error status to the NDE DHS.
PCF PSFDriver script log fileExecutable program log fileResource input file
main_npr It converts products between NetCDF4 and BUFR formats. For NC2BUFR, it reads BUFR table and NetCDF4 files listed in the resource file, and writes out BUFR file. For BUFR2NC, it reads BUFR table and BUFR file, and writes out NetCDF4 files.
For NC2BUFR: BUFR table, Geo NetCDF4 file, SDR NetCDF4For BUFR2NC: BUFR table and BUFR file.
For NC2BUFR: BUFR file.For BUFR2NC: NetCDF4 Geo and SDR files
140
Test Data Description
· Simulated near real-time NPP 72 hours data (09/06/2010 – 09/08/2010)»Random choose one granule file for ATMS, VIIRS
M-band, and VIIRS I-band SDR data.»According to the time mark of the chosen granule,
choose the matched geolocation data.»One VIIRS granule length: 85.7856 seconds.»One ATMS length: 32 seconds.»One CrIS granule length: 32 seconds.
141
BUFR Table Description
· All the BUFR tables are discussed among the user communities including NCEP, ECMWF, EUMETSAT and UK Met Office, and get WMO approved.
· The BUFR tables have the required variables and descriptors.
· The delayed replication factor is applied for the channel data, which adds the flexibility to choose the channel subset.
· VIIRS M-band (16 channels), I-band (5 channels) and DNB Band (1 channel) share one BUFR table.
142
Test Configuration· Each item in the test configuration has been placed in the project
baseline under configuration control
· The Perl script can be run as it is.
· The Fortran 90 and C code must be compiled.
· The entire NC2BUFR system is compiled with a single Makefile. It can be run using gmake at AIX machine /usr/bin/gmake.
· After gmake has run, the user must have main_npr copied to /disk2/pub/czhang/CrISOPS/Common_Bin, or update the NPR.pl.PCF file to use the correct OPS_BIN directory.
143
Test Sequence/Results· Test Step 1: Changed into the test directory on orbit002b:
» /disk2/pub/ysong/test_code/ATMS
· Test Step 2: Verified that the PCF contents (NPR.pl.PCF) were created by following the NDE Algorithm Delivery Standards (Appendix A – NDE DHS Product Generation Algorithm Interface) by dumping the contents of the file:
job_coverage_start=201009072208471job_coverage_end=201009072209185PROD_TYPE=ATMSCONVERSION=NC2BUFRNPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR-GEO/GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR/SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.h5NPR_INPUT=/net/orbit001x/disk002/pub/npp-p72/ATMS-TDR/TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.h5BUFR_TABLE=/disk2/pub/czhang/NPROPS/BUFR/ATMS_BUFR_TableOPS_BIN=/disk2/pub/ysong/code/main
144
Test Sequence/Results· Test Step 3: NPR.pl can check the presence of NPR.pl.PCF:
· Test Step 4: NPR.pl can read NPR.pl.PCF:
sysopen(PCF, "NPR.pl.PCF",O_RDONLY) or $rc = 1; if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: failed to open NPR.pl.PCF. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }
@Line_List = <PCF>; close (PCF);
if( ! -e "NPR.pl.PCF" ) { print "Error in ${PROGRAM}.pl: NPR.pl.PCF does not exist. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }
145
Test Sequence/Results· Test Step 5: Ran the driver script as:
» perl -w /disk2/pub/ysong/test_code/ NPR.pl 2>&1
· Test Step 6: Verified that the PSF contents (NPR.pl.PSF) were created indeed in accordance with the NDE Algorithm Delivery Standards (Appendix A – NDE DHS Product Generation Algorithm Interface) by dumping the contents of the file:
The file contains the output file name only when the program completes successfully. Otherwise, it is empty.
orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>ls NPR.pl.PSFNPR.pl.PSForbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>cat NPR.pl.PSFATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221457390.bufrorbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>
146
Test Sequence/Results
· Test Step 7: Each of the 5 different BUFR files can be produced successfully: ATMS, CrIS 399, CrIS 1305, VIIRS I5, and VIIRS M4. An ls -lrt in the test directory can verify that the output files were generated by this run of the script.
Here is ATMS output for a run at 11:49 on Sept. 22, 2011:
orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>orbit002b(ysong):/disk2/pub/ysong/test_code/ATMS>ls -lrttotal 0 -rw-r--r-- 1 ysong orbit 597 Sep 22 11:38 NPR.pl.PCF-rw-r--r-- 1 ysong orbit 366 Sep 22 11:49 npr.filenames-rwxr-xr-x 1 ysong orbit 157600 Sep 22 11:49 TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 171152 Sep 22 11:49 SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 157784 Sep 22 11:49 GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc-rw-r--r-- 1 ysong orbit 1975 Sep 22 11:49 NPR.pl.log-rw-r--r-- 1 ysong orbit 70 Sep 22 11:49 NPR.pl.PSF-rw-r--r-- 1 ysong orbit 127072 Sep 22 11:49 ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr
147
Test Sequence/Results
· Here is CrIS 1305 Channels output for a run at 12:47 on Sept. 22, 2011:
· This is CrIS 399 Channels output for a run at 14:54 on Oct. 21, 2011:
orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 5871947 Aug 25 15:57 NUCAPS_ALL_20110825_1227240_1227538.nc-rw-r--r-- 1 ysong orbit 245 Aug 26 10:36 NPR.pl.PCF-rw-r--r-- 1 ysong orbit 178 Sep 22 12:47 npr.filenames-rw-r--r-- 1 ysong orbit 2881664 Sep 22 12:47 NUCAPS-C1305_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr-rw-r--r-- 1 ysong orbit 727 Sep 22 12:47 NPR.pl.log-rw-r--r-- 1 ysong orbit 78 Sep 22 12:47 NPR.pl.PSF
orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS399>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/CrIS399>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 1898843 Aug 25 15:57 NUCAPS_C0300_ALLFOVS_20110825_1241540_1242237.nc-rw-r--r-- 1 ysong orbit 255 Aug 26 10:38 NPR.pl.PCF-rw-r--r-- 1 ysong orbit 188 Oct 21 14:54 npr.filenames-rw-r--r-- 1 ysong orbit 27900 Oct 21 14:54 fort.30-rw-r--r-- 1 ysong orbit 803288 Oct 21 14:54 NUCAPS-C0399_v1r0_npp_s201108251241540_e201108251242237_c201110211854110.bufr-rw-r--r-- 1 ysong orbit 790 Oct 21 14:54 NPR.pl.log-rw-r--r-- 1 ysong orbit 78 Oct 21 14:54 NPR.pl.PSF
148
Test Sequence/Results
· Here is VIIRS M4 output for a run at 12:53 on Sept. 22, 2011:
orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_M4>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1 orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_M4>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 904 Aug 22 15:39 NPR.pl.PCF-rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM12_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089183_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 81187816 Sep 22 12:51 GMODO_npp_d20100908_t2222327_e2223569_b00042_c20110408003232093783_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM15_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089860_noaa_ops.nc-rwxr-xr-x 1 ysong orbit 22190144 Sep 22 12:51 SVM13_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089418_noaa_ops.nc-rw-r--r-- 1 ysong orbit 537 Sep 22 12:51 npr.filenames-rwxr-xr-x 1 ysong orbit 12365008 Sep 22 12:51 SVM16_npp_d20100908_t2222327_e2223569_b00042_c20110408003232090078_noaa_ops.nc-rw-r--r-- 1 ysong orbit 33499512 Sep 22 12:53 VIIRSM4_v1r0_npp_s201009082222327_e201009082223569_c201109221651090.bufr-rw-r--r-- 1 ysong orbit 2661 Sep 22 12:53 NPR.pl.log-rw-r--r-- 1 ysong orbit 73 Sep 22 12:53 NPR.pl.PSF
149
Test Sequence/Results
· VIIRS I5 output for a run at 12:58 on Sept. 22, 2011 is: orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_I5>perl -w /disk2/pub/ysong/test_code/NPR.pl 2>&1orbit002b(ysong):/disk2/pub/ysong/test_code/VIIRS_I5>ls -lrttotal 0-rw-r--r-- 1 ysong orbit 489 Aug 24 16:35 NPR.pl.PCF-rwxr-xr-x 1 ysong orbit 324489816 Sep 22 12:50 GIMGO_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654742_noaa_ops.nc-rw-r--r-- 1 ysong orbit 300 Sep 22 12:50 npr.filenames-rwxr-xr-x 1 ysong orbit 49230656 Sep 22 12:50 SVI05_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654462_noaa_ops.nc-rw-r--r-- 1 ysong orbit 118547320 Sep 22 12:58 VIIRSI5_v1r0_npp_s201009082215257_e201009082216502_c201109221650280.bufr-rw-r--r-- 1 ysong orbit 2032 Sep 22 12:58 NPR.pl.log-rw-r--r-- 1 ysong orbit 73 Sep 22 12:58 NPR.pl.PSF
150
Test Sequence/Results
· The existence of the NetCDF4 files:
demonstrates that h5augjpss was able to convert HDF5 file to the readable NetCDF4 files.
· The existence of the output file:
demonstrates that the script NPR.pl was able to:
» Read the PCF » Create a resource file for the executable (npr.filenames) » Run the executable main program main_npr.
GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.ncSATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.ncTATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.nc
ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr
151
Test Sequence/Results
· Test Step 8: Dump HDF5 file and its NetCDF4 file. The same values within precision show that the utility “h5augjpss” is able to convert HDF5 to NetCDF4.» Part of Latitude data in
/net/orbit001x/disk002/pub/npp-p72/ATMS-SDR-GEO/GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5
» Part of Latitude data in GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.nc
DATASET "Latitude" { DATATYPE H5T_IEEE_F32BE DATASPACE SIMPLE { ( 12, 96 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) } DATA { (0,0): -73.1603, -73.2373, -73.2889, -73.3205, -73.3363, (0,5): -73.3392, -73.3319, -73.3162, -73.2936, -73.2653, -73.232, (0,11): -73.195, -73.1546, -73.1115, -73.066, -73.0186, -72.9694, (0,17): -72.9189, -72.8671, -72.8142, -72.7604, -72.7058, (0,22): -72.6505, -72.5946, -72.5381, -72.4809, -72.4234,
Latitude = -73.16026, -73.23733, -73.28891, -73.32051, -73.33625, -73.33925, -73.33195, -73.31623, -73.2936, -73.26526, -73.23201, -73.19497, -73.15462, -73.1115, -73.06602, -73.01857, -72.96943, -72.91886, -72.86705, -72.81418, -72.76039, -72.7058, -72.65051, -72.5946, -72.53812, -72.48086, -72.42341, -72.36552, -72.30724, -72.24854, -72.18947, -72.13002, -72.07019, -72.00996, -71.94936, -71.88834,
152
Test Sequence/Results· Test Step 9: Dumped the Contents of the internal
resource file (npr.filenames). This shows that NPR.pl was able to read and parse the PCF file contents to the resource file, and use internal readable NetCDF4 files for input to main program main_npr.
ATMSNC2BUFRGATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.ncSATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.ncTATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.ncATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr/disk2/pub/czhang/NPROPS/BUFR/ATMS_BUFR_Table
153
Test Sequence/Results· Test Step 10: Dumped the contents of the log file (NPR.pl.log) to show
that everything ran correctly. There were no errors in the script itself, any of the system calls, the main program, its subroutines and BUFR library calls. This log file contained the starting and ending times indicating everything completed in 1 seconds of clock time. The CPU for the main program is 0.4 second for one ATMS granule data.
Starting at Thu Sep 22 11:49:10 EDT 2011 main_npr is now starting. Starting ATMS_netcdf_to_bufr Output_FileName=ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr Starting ATMS_write_bufr
+++++++++++++++++BUFR ARCHIVE LIBRARY++++++++++++++++++++ BUFRLIB: MAXOUT - THE RECORD LENGTH OF ALL BUFR MESSAGES CREATED FROM THIS POINT ON IS BEING CHANGED FROM 10000 TO 20000 +++++++++++++++++BUFR ARCHIVE LIBRARY++++++++++++++++++++
Closing BUFR file. Finishing ATMS_write_bufr Finishing ATMS_netcdf_to_bufr main_npr is now finished. CPU time for the whole program is: 0.400000000000000022Ending at Thu Sep 22 11:49:11 EDT 2011
154
Test Sequence/Results· Test Step 11: The BUFR tables have the required variables and
descriptors. Some special requirements, such as negative radiance, are kept. Example of CrIS BUFR table..------------------------------------------------------------------------------.| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- ||------------------------------------------------------------------------------|| MNEMONIC | NUMBER | DESCRIPTION ||----------|--------|----------------------------------------------------------|| | | || NC021202 | A10060 | MSG TYPE 021-202 CrIS radiance data || | | || YYMMDD | 301011 | Date -- year, month, day || HHMM | 301012 | Time -- hour, minute || LTLONH | 301021 | High accuracy latitude/longitude position || LOCPLAT | 304030 | Location of platform (satellite) sequence || BCFQFSQ | 350200 | NPP CrIS band calibration & f-o-v quality flag sequence || CRCHN | 350201 | NPP CrIS channel data || | | || SAID | 001007 | Satellite identifier || OGCE | 001033 | Identification of originating/generating center || SIID | 002019 | Satellite instruments || SCLF | 002020 | Satellite classification || RDTF | 002165 | Radiance type flags || YEAR | 004001 | Year || MNTH | 004002 | Month || DAYS | 004003 | Day || HOUR | 004004 | Hour || MINU | 004005 | Minute || SECO | 004006 | Second || CLATH | 005001 | Latitude (high accuracy) || BEARAZ | 005021 | Bearing or azimuth || SOLAZI | 005022 | Solar azimuth || ORBN | 005040 | Orbit number || SLNM | 005041 | Scan line number || CHNM | 005042 | Channel number || FOVN | 005043 | Field of view number || FORN | 005045 | Field of regard number |
155
Example of BUFR table (CrIS) Continued
| CLONH | 006001 | Longitude (high accuracy) || WVNM | 006029 | Wave number || HMSL | 007002 | Height or altitude || SAZA | 007024 | Satellite zenith angle || SOZA | 007025 | Solar zenith angle || LSQL | 008012 | Land/Sea qualifier || STKO | 008075 | Ascending/Descending orbit qualifier || TOBD | 008076 | Type of band || HOLS | 010001 | Height of land surface || PDNP | 010031 | In dir. of North Pole, distance from the Earth's center || SRAD | 014044 | Channel radiance || TOCC | 020010 | Cloud cover (total) || HOCT | 020014 | Height of top of cloud || ALFR | 021166 | Land fraction || STCH | 025140 | Start channel || ENCH | 025141 | End channel || PD00 | 027031 | In direction of 0 deg E, distance from Earth's center || PD90 | 028031 | In direction of 90 deg E, distance from Earth's center || QMRKH | 033003 | Quality information || NSQF | 033075 | Scan-level quality flags || NCQF | 033076 | Calibration quality flags || NFQF | 033077 | Field of view quality flags || NGQI | 033078 | Geolocation quality || | | ||------------------------------------------------------------------------------|| MNEMONIC | SEQUENCE ||----------|-------------------------------------------------------------------|| NC021202 | SAID OGCE SIID SCLF YYMMDD HHMM || NC021202 | 207003 SECO 207000 LOCPLAT LTLONH SAZA BEARAZ || NC021202 | SOZA SOLAZI STKO 201133 SLNM 201000 FORN || NC021202 | FOVN ORBN HOLS 201129 HMSL 201000 || NC021202 | 202127 201125 ALFR 201000 202000 LSQL TOCC || NC021202 | HOCT RDTF NSQF "BCFQFSQ"3 || NC021202 | TOBD NGQI QMRKH (CRCHN) || | || YYMMDD | YEAR MNTH DAYS || HHMM | HOUR MINU |
156
Example of BUFR table (CrIS) Continued
| LTLONH | CLATH CLONH || LOCPLAT | PD00 PD90 PDNP || BCFQFSQ | TOBD WVNM WVNM STCH ENCH NCQF NFQF || CRCHN | 201133 CHNM 201000 SRAD || | ||------------------------------------------------------------------------------|| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------||----------|------|-------------|-----|--------------------------|-------------|| | | | | |-------------|| SAID | 0 | 0 | 10 | Code table |-------------|| OGCE | 0 | 0 | 8 | Code table |-------------|| SIID | 0 | 0 | 11 | Code table |-------------|| SCLF | 0 | 0 | 9 | Code table |-------------|| RDTF | 0 | 0 | 15 | Flag table |-------------|| YEAR | 0 | 0 | 12 | Year |-------------|| MNTH | 0 | 0 | 4 | Month |-------------|| DAYS | 0 | 0 | 6 | Day |-------------|| HOUR | 0 | 0 | 5 | Hour |-------------|| MINU | 0 | 0 | 6 | Minute |-------------|| SECO | 0 | 0 | 6 | Second |-------------|| CLATH | 5 | -9000000 | 25 | Degree |-------------|| BEARAZ | 2 | 0 | 16 | Degree true |-------------|| SOLAZI | 2 | 0 | 16 | Degree true |-------------|| ORBN | 0 | 0 | 24 | Numeric |-------------|| SLNM | 0 | 0 | 8 | Numeric |-------------|| CHNM | 0 | 0 | 6 | Numeric |-------------|| FOVN | 0 | 0 | 8 | Numeric |-------------|| FORN | 0 | 0 | 8 | Numeric |-------------|| CLONH | 5 | -18000000 | 26 | Degree |-------------|| WVNM | 1 | 0 | 22 | m**-1 |-------------|| HMSL | -1 | -40 | 16 | m |-------------|| SAZA | 2 | -9000 | 15 | Degree |-------------|| SOZA | 2 | -9000 | 15 | Degree |-------------|| LSQL | 0 | 0 | 2 | Code table |-------------|| STKO | 0 | 0 | 2 | Code table |-------------|| TOBD | 0 | 0 | 6 | Code table |-------------|| HOLS | 0 | -400 | 15 | m |-------------|| PDNP | 2 | -1073741824 | 31 | m |-------------|| SRAD | 7 | -100000 | 22 | W m**-2 sr**-1 cm**-1 |-------------|
157
Example of BUFR table (CrIS) Continued
| TOCC | 0 | 0 | 7 | % |-------------|| HOCT | -1 | -40 | 11 | m |-------------|| ALFR | 3 | 0 | 10 | Numeric |-------------|| STCH | 0 | 0 | 14 | Numeric |-------------|| ENCH | 0 | 0 | 14 | Numeric |-------------|| PD00 | 2 | -1073741824 | 31 | m |-------------|| PD90 | 2 | -1073741824 | 31 | m |-------------|| QMRKH | 0 | 0 | 3 | Code table |-------------|| NSQF | 0 | 0 | 13 | Flag table |-------------|| NCQF | 0 | 0 | 9 | Flag table |-------------|| NFQF | 0 | 0 | 19 | Flag table |-------------|| NGQI | 0 | 0 | 4 | Code table |-------------|| | | | | |-------------|`------------------------------------------------------------------------------'
· The reference value of -100000 of Channel radiance (SRAD 0-14-044) allows to keep negative radiance in the CrIS BUFR file.
| SRAD | 7 | -100000 | 22 | W m**-2 sr**-1 cm**-1 |-------------|
158
Test Sequence/Results· Test Step 12: In BUFR table, the delayed replication factor is applied for
the channel data. It adds the flexibility to apply the BUFR table to the data subset, such as delayed replication for channel data in CrIS BUFR table
Entry number Meaning
1 04 000 Delayed replication of 4 descriptors
0 31 002 Extended delayed descriptor replication factor
2 01 133 Increase bit width0 05 042 Channel number2 01 000 Cancel increase bit width
0 14 044 Channel radiance
159
Test Sequence/Results· Test Step 13: The delayed replication is
implemented in the program code.
» Part of code in subroutine nucape_write_bufr.f90 for CrIS.
» The argument “NUM_CRIS” in the NCEP BUFR library call “CALL DRFINI(LUBFR,NUM_CRIS,1,'(CRCHN)')” tells the system there are how many replication of CrIS channel data.
!! Open a BUFR message!
CALL OPENMB(LUBFR,SUBSET,IDATE) CALL DRFINI(LUBFR,NUM_CRIS,1,'(CRCHN)') CALL UFBSEQ(LUBFR,ARR,ARRDIM,1,IRET,SUBSET) CALL WRITCP(LUBFR)
160
Test Sequence/Results· Test Step 14: The output BUFR file name follow the NDE file name
convention defined in the NDE document called "Algorithm Delivery Standards, Integration, and Test_V1.3.docx”.
» Data Product Short Name is ATMS » The version of this product, which is 1.0 (v1r0). » The platform from which this file was created is the NPP satellite
(npp). » Start date and time of the data is September 07, 2010 at 22:08:47.1
UTC (s201009072208471).» End date and time of the data is September 07, 2010 at 22:09:18.5
UTC (e201009072209185).» This output product file was created on September 22, 2011 at
15:49:10.0 UTC (c201109221549100). » The file is a BUFR file (.bufr extension).
ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr
161
Test Sequence/Results· Test Step 15: The created BUFR file can be read back and are
comparable with the original NetCDF4 file. » Part of channel radiances that are converted back from its BUFR file (
» Part of channel radiances in the original NetCDF4 file.
· The comparable values in the two files demonstrate that the system can convert from NetCDF4 to BUFR and then back to NetCDF4
CrIS_Radiances (unit: mW/m2/cm-1/sr ) = 45.4617, 52.7766, 69.9809, 62.8292, 54.4361, 47.8841, 44.4845, 45.2233, 47.1264, 41.4511, 43.0444, 45.3617, 43.0024, 42.4175, 39.3311, 41.1938, 40.666, 39.6869, 39.0525, 40.0077, 40.4335, 39.883, 41.3726, 42.5681,
CrIS_Radiances (unit: mW/m2/cm-1/sr ) = 45.46172, 52.7766, 69.98087, 62.82917, 54.43607, 47.88414, 44.48447, 45.22329, 47.1264, 41.45107, 43.04439, 45.3617, 43.00241, 42.41754, 39.3311, 41.19378, 40.666, 39.68685, 39.05251, 40.00766, 40.43351,
162
Test Sequence/Results· Test Step 16: The created BUFR file can also be read with the ECMWF
BUFR library. This indicates that the products are readable among global user communities.» Part of values read from the BUFR file with ECMWF BUFR library.
57 EXTENDED DELAYED DESCRIPTOR REPL 0.39900000000000E+003 NUMERIC 58 CHANNEL NUMBER 0.27000000000000E+002 NUMERIC 59 CHANNEL RADIANCE 0.45461700000000E-001 (W/M**2)*(1/SR)*CM 60 CHANNEL NUMBER 0.28000000000000E+002 NUMERIC 61 CHANNEL RADIANCE 0.52776600000000E-001 (W/M**2)*(1/SR)*CM 62 CHANNEL NUMBER 0.31000000000000E+002 NUMERIC 63 CHANNEL RADIANCE 0.69980900000000E-001 (W/M**2)*(1/SR)*CM 64 CHANNEL NUMBER 0.32000000000000E+002 NUMERIC 65 CHANNEL RADIANCE 0.62829200000000E-001 (W/M**2)*(1/SR)*CM 66 CHANNEL NUMBER 0.33000000000000E+002 NUMERIC 67 CHANNEL RADIANCE 0.54436100000000E-001 (W/M**2)*(1/SR)*CM 68 CHANNEL NUMBER 0.37000000000000E+002 NUMERIC 69 CHANNEL RADIANCE 0.47884100000000E-001 (W/M**2)*(1/SR)*CM 70 CHANNEL NUMBER 0.49000000000000E+002 NUMERIC 71 CHANNEL RADIANCE 0.44484500000000E-001 (W/M**2)*(1/SR)*CM 72 CHANNEL NUMBER 0.51000000000000E+002 NUMERIC 73 CHANNEL RADIANCE 0.45223300000000E-001 (W/M**2)*(1/SR)*CM 74 CHANNEL NUMBER 0.53000000000000E+002 NUMERIC 75 CHANNEL RADIANCE 0.47126400000000E-001 (W/M**2)*(1/SR)*CM 76 CHANNEL NUMBER 0.59000000000000E+002 NUMERIC 77 CHANNEL RADIANCE 0.41451100000000E-001 (W/M**2)*(1/SR)*CM 78 CHANNEL NUMBER 0.61000000000000E+002 NUMERIC 79 CHANNEL RADIANCE 0.43044400000000E-001 (W/M**2)*(1/SR)*CM
163
Test Sequence/Results· Test Step 17: Error checks execute in script. In the script file (NPR.pl), all
system calls are check and exit gracefully with an error message when a problem occurs.
» Part of script program NPR.pl. $rc=system("cp $input_file $nc_name"); if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: the copy data returned an error. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }
$rc=system("$h5augjpss -o1 -o4 $nc_name"); if( $rc != 0 ) { print "Error in ${PROGRAM}.pl: h5augjpss returned an error. "; print "${PROGRAM}.pl will now exit.\n"; $STATUS_NPR_PRODUCT = "NO"; &make_psf(); exit 1; }
164
Test Sequence/Results· Test Step 18: Error checks process in the main NC2BUFR system. In the
main program (main_npr.f90) and its subroutines, all system calls are check and exit gracefully with an error message when a problem occurs. » Part of program main_npr.f90
PRINT*,'main_npr is now starting.' call cpu_time(t1) Unit_Number_Resource_File = Get_Lun(); OPEN (UNIT=Unit_Number_Resource_File, File = "npr.filenames", & Status="OLD", ACTION = "READ", IOSTAT=Open_Status)
IF(Open_Status /= 0)THEN CALL error_messaging('main_npr', & 'Failed to open npr.filenames', & FATAL) ENDIF
!! Read the input resource file.!
READ(Unit_Number_Resource_File,FMT='(a)',IOSTAT=Read_Status)Product_Type
IF(Read_Status /= 0)THEN CALL error_messaging('main_npr', & 'Failed to read npr.filenames', & FATAL) ENDIF
165
Test Sequence/Results· Test Step 19: Run times are very short for converting the HDF5 ATMS
and VIIRS files using the h5augjpss tool. It is less than 1 seconds for ATMS, 4 seconds for VIIRS 4 M-bands, and 7 seconds for VIIRS 1 I-band.
» Part of NPR.pl.log for ATMS
» Part of NPR.pl.log for VIIRS 4 M-bands
» Part of NPR.pl.log for VIIRS 5 I-bands
start_h5augjpss=1316795761, end_h5augjpss=1316795761, h5augjpss_time=0
start_h5augjpss=1316803047, end_h5augjpss=1316803051, h5augjpss_time=4
start_h5augjpss=1316803089, end_h5augjpss=1316803096, h5augjpss_time=7
166
Test Sequence/Results· Test Step 20: CPU times for generating BUFR file. It is short for CrIS and
ATMS, but it takes long to convert VIIRS SDR files to BUFR files. » It took 0.4 seconds CPU time to process one granule ATMS.
» It took 6.24 seconds CPU time to process one granule CrIS all channels.
» It took 3.62 seconds CPU time to process one granule CrIS 399 channels.
» It took 146.96 seconds CPU time to process one granule VIIRS 4 M-bands.
» It took 437.56 seconds CPU time to process one granule VIIRS 1 I-bands.
CPU time for the whole program is: 0.400000000000000022
CPU time for the whole program is: 6.23999999999999932
CPU time for the whole program is: 3.61999999999999966
CPU time for the whole program is: 146.960000000000008
CPU time for the whole program is: 437.560000000000002
167
Test Sequence/Results· Test Step 21: BUFR file sizes of the created BUFR files.
» Size of one granule all channel CrIS BUFR file is 2.882 mb. The size of whole day BUFR files is: 2.882*(86400/32)=7781.4 mb=7.781 gb
» Size of one granule 399 channels CrIS BUFR file is 0.803 mb. The size of whole day CrIS 399 channels BUFR files is: 0.803*(86400/32)=2168.1 mb = 2.168gb.
» Size of one granule ATMS BUFR file is 0.127 mb. The size of whole day BUFR files is: 0.127*(86400/32)=342.9 mb=0.343 gb.
» Size of one granule VIIRS 4 M-bands BUFR file is 33.5 mb. The size of whole day BUFR files is: 33.5*(86400/85.7856)=33739.9 mb=33.7 gb
» Size of one granule VIIRS 1 I-bands BUFR file is 118.5 mb. The size of whole day BUFR files is: 118.5*(86400/85.7856)=119348.7 mb=119.3 gb
168
Test Risks· Risk #1: The required input data are NetCDF4, but the SDR data are in HDF5.
The NDE converter from HDF5 to NetCDF4 was not ready at the time of testing. Also, NDE’s solution would’ve involved bundling all the SDRs into one file and changing variable names so the converter’s readers would’ve needed to change.
· Risk Assessment: Low· Impact: This unavailable converter may cause this system development delay
and affect the test.· Likelihood: Low· Mitigation:
» The HDF group utility h5augjpss is available from July 2011, and we have already verified its results. This utility is being used for this test and will be the delivered solution.
» To speed up the development, we implemented our own converter from hdf5 to NetCDF4. This is an alternative solution that is available.
· Status: Close.
169
Test Risks
· Risk #2: It took 146.96 seconds CPU time to process one 85.7856 seconds granule VIIRS 4 M-bands, and 437.56 seconds for VIIRS 1 I-bands. It is too slow.
· Risk Assessment: Low· Impact: The long run times may delay the products.· Likelihood: Low· Mitigation:
» After talking with NDE, we found that NDE has enough CPUs for current processing needs (96 AIX CPUs).
» VIIRS TIM has reduced the required VIIRS channel set, but the processing time may be an issue in the future if the requirements are expanded.
· Status: Close. We can close this for Phase 1 SDR delivery, but it may be an issue later in the future if requirements change.
Summary & Conclusions· NC2BUFR system has been test step by step, and it has
passes all the tests.· The main program and script check the status of all
system calls and exit gracefully with an error message when a problem occurs.
· This system can convert from NetCDF4 to BUFR and then back to NetCDF4.
· The BUFR tables have the required variables and descriptors.
· ATMS, CrIS 399, CrIS 1305, VIIRS I5, and VIIRS M4 BUFR files can be produced successfully.
· Output BUFR files follow the NDE file name convention.170
171
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
172
Section 7 –Delivered Algorithm
Package
Presented by
Thomas KingRiverside
173
NDE DAP Requirements
· 1. Delivery Memo· 2. README text file· 3. Science algorithm source code, including make files, and scripts. · 4. Input test data and resulting output data.· 5. Static coefficient files and/or look-up table files if applicable· 6. Description of the processing steps in the form of a data flow diagram consistent with SA software
architecture.· 7. Description of all input data files and any intermediate data files produced during SA execution. · 8. Description of output data files contents, requirements (latency and quality), storage requirements,
and format. · 9. Description of product quality monitoring information (exception handling, exit codes and quality
flags along with their meanings). · 10. Detailed algorithm test plan (description, procedures, product quality requirements, and results). · 11. Estimates of computer and software storage resources required for execution. Include software
storage requirements for SA code, static data files, and computer resources such as wall clock time, CPU time, memory, page faults, and disk space.
· 12. Algorithm Theoretical Basis Document (ATBD). This may be a specific pointer (or hyperlink) to the document.
174
Toolkit DAP Requirements
· Delivery memo will contain:» Point(s) of contact (POC) for questions specific to the algorithm
(include name, telephone, and e-mail address). At a minimum, include algorithm developer and PAL.
» Date of Current DAP Release» Purpose of the delivery (e.g. an initial release, modification)» Brief description of package (e.g. “DAP Version X represents the initial
delivery of the…”, see Appendix B for a more detailed example)» List of DAP contents» Description of problem(s) resolved, if any, and method of resolution.» Description of significant changes from previous version, if any.» List of known remaining updates/issues.
175
Toolkit DAP Requirements
· The README text file in the DAP must contain:» DAP version number.» Date of current DAP release » Target configuration for SA source code execution. » List of all pointers (or hyperlinks) to reference documents (DAP
contents 6-12 above). » Detailed description of how to compile the source code in the NDE
SADIE environment. » List of expected compiler warnings.» Other pertinent information as judged by the algorithm developer(s)
(compiler settings, etc.).» Description of PCF and PSF file. » Production Rule Definitions
176
N4RT DAP Contents
· The DAP will be delivered to SADIE and will consist of a gzip’d tar file that will extract 4 main directories into whatever is the current directory of the DAP tar file. The directories are: » Source
– All Fortran source code for N4RT (4 MB)– BUFRLIB source (3.6 MB)– h5augjpss source (96 MB)
» OPS– All scripts (0.012 MB)– BUFR tables (0.032 MB)– Compiled (executable) (22 MB)
» DATA– ATMS (1 MB)– CrIS_C0399 (2.9 MB)– CrIS_C1305 (8.6 MB)– VIIRS_I5 (512 MB)– VIIRS_M4 (193 MB)
» DOCS– All SPSRB and NDE documentation (17 MB)
· Total size of DAP will be approximately 860 MB
177
Toolkit Source Contents
Code Language Number of Files Number of Lines
N4RT Fortran 90 325 78551
N4RT Perl 1 424
BUFRLIB 4.0 Fortran 77/C 218 27472
h5augjpss source 1.0.0 C 7 1983
178
Toolkit OPS Contents
Content Files Description
Scripts NPR.pl The driver script invoked by the NDE DHS.
BUFR TablesATMS_BUFR_TableCrIS_BUFR_Table
VIIRS_BUFR_TableThe ASCII BUFR tables input to the Fortran executable
Compiled Codemain_nprh5augjpss
The executables
179
Toolkit DATA Contents (Input
Files)Files Description
NUCAPS_C0300_ALLFOVS_20110825_1227240_1227538.nc NUCAPS 399-channel input file
NUCAPS_ALL_20110825_1227240_1227538.nc NUCAPS 1305-channel input file
TATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994562_noaa_ops.h5SATMS_npp_d20100907_t2208471_e2209185_b00027_c20110406231119994220_noaa_ops.h5GATMO_npp_d20100907_t2208471_e2209185_b00027_c20110406231119993808_noaa_ops.h5
ATMS input HDF5 input files
GIMGO_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654742_noaa_ops.h5SVI05_npp_d20100908_t2215257_e2216502_b00042_c20110408002710654462_noaa_ops.h5
VIIRS I5 HDF5 input files
GMODO_npp_d20100908_t2222327_e2223569_b00042_c20110408003232093783_noaa_ops.h5SVM12_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089183_noaa_ops.h5SVM15_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089860_noaa_ops.h5SVM13_npp_d20100908_t2222327_e2223569_b00042_c20110408003232089418_noaa_ops.h5SVM16_npp_d20100908_t2222327_e2223569_b00042_c20110408003232090078_noaa_ops.h5
VIIRS M4 HDF5 input files
180
Toolkit DATA Contents (Output
Files)
Files Description
ATMS_v1r0_npp_s201009072208471_e201009072209185_c201109221549100.bufr ATMS output BUFR file
C0399_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr CrIS 399-channel BUFR file
C1305_v1r0_npp_s201108251227240_e201108251227538_c201109221647010.bufr CrIS 1305-channel BUFR file
VIIRSI5_v1r0_npp_s201009082215257_e201009082216502_c201109221650280.bufr VIIRS I5 BUFR file
VIIRSM4_v1r0_npp_s201009082222327_e201009082223569_c201109221651090.bufr VIIRS M4 BUFR file
181
Toolkit DOCS Contents
Code Document Type Description
System Maintenance Manual SPSRB
Describes everything in the N4RT system: all input/output/intermediate data, LUTs, and overall software architecture (processing steps). Also describes
production rules, error handling, quality monitoring, computer resource requirements.
External Users Manual SPSRB Describes formats of all output products and everything required to use them.
Test Readiness Document STAR EPL The slide package showing the test plan and the results of all the units tests. Also describes all input/output test data.
README NDE README file for NDE DAP
182
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
183
Section 8 –Risks and Actions
Summary
Presented by
Thomas KingRiverside
184
Risk and Actions
· The status of risks from the PDR and CDR Report were reported earlier in this review
· Only the remaining open risks will be summarized here.
· The status of any new risks since the CDR will be reported here as well.
185
Risk Summary· Risk 3: There are small variations in the types of platforms
and the versions of the compilers» Mitigation: Reformatting Toolkit developers will work with NDE during
system tests in the integration and production phase to ensure that those results match the results from the units. The delivery of prototype DAPs should help to resolve these issues early on in development.
· Risk 4: Data format translation involves some unit conversion and possible reduction of precision (significant digits)» Mitigation: Document data manipulations in the SMM which will be
included in the NDE Delivered Algorithm Package. This will server as a replacement for an ATBD.
186
Risk Summary· Risk 6: Risk on NDVI and snow mask. Have not yet
demonstrated ability to encode GRIB2 files.» Mitigation: NDVI and snow mask will not be produced. However, GVF
will need to be in GRIB2 format, but that’s something that will not be a capability of the toolkit until Phase 1 EDR DAP. This capability will be demonstrated at the Phase 1 EDR & Phase 2 EDR products TRR scheduled for 1/23/2012.
· Risk 18: ESPC may not have the BUFR expertise to maintain the toolkit.» Mitigation: OSPO is getting FY12 funding to support the transition and
maintenance from NDE on the OSPO side.
187
New Risks· Risk 20: The required input data are NetCDF4, but the SDR data are in HDF5.
The NDE converter from HDF5 to NetCDF4 was not ready at the time of testing. Also, NDE’s solution would’ve involved bundling all the SDRs into one file and changing variable names so the converter’s readers would’ve needed to change.
· Risk Assessment: Low· Impact: This unavailable converter may cause this system development delay
and affect the test.· Likelihood: Low· Mitigation:
» The HDF group utility h5augjpss is available from July 2011, and we have already verified its results. This utility is being used for this test and will be the delivered solution.
» To speed up the development, we implemented our own converter from hdf5 to NetCDF4. This is an alternative solution that is available.
· Status: Closed.
188
New Risks
· Risk 21: It took 146.96 seconds CPU time to process one 85.7856 seconds granule VIIRS 4 M-bands, and 437.56 seconds for VIIRS 1 I-bands. It is too slow.
· Risk Assessment: Low· Impact: The long run times may delay the products.· Likelihood: Low· Mitigation:
» After talking with NDE, we found that NDE has enough CPUs for current processing needs (96 AIX CPUs).
» VIIRS TIM has reduced the required VIIRS channel set, but the processing time may be an issue in the future if the requirements are expanded.
· Status: Closed. We can close this for Phase 1 SDR delivery, but it may be an issue later in the future if requirements change.
189
New Risks· Risk 22: NDE needs to modify their contract to support completion of
the SPSRB documents delivered by the NDE product teams. This is needed for a timely and smooth transition to OSPO. This is a project-level risk because there is concern over whether this can be done before NDE system is declared operational.
· Assessment: Low· Impact: OSPO may not have sufficient documentation of the
delivered NDE system. · Likelihood: Moderate· Mitigation: Geof Goodrum is modifying the NDE contract to include
this documentation.· Status: Open.
190
Risks and Actions Summary
· There are 11 risks that carry over from PDR/CDR:
· We recommend closing 8 of these risks: » 4 risks will remain open.
· There are 3 new risks between CDR and TRR» 2 of these can be closed.
191
Review Outline· Introduction· CDR review report/actions· Requirements· Quality Assurance· Software architecture · Unit Tests· Delivered Algorithm Package· Risks and Actions Summary· Summary and Conclusions
192
Section 9 –Summary and
Conclusions
Presented by
Walter WolfNOAA/NESDIS/STAR
193
· The Project Mission and Schedule has been reviewed.
· The CDR Report has been reviewed.
· The Project Requirements have been reviewed.
· Plans for quality assurance have been reviewed.
· Software architecture, hardware, data, and interfaces have been reviewed.
· Test plans and results have been documented and presented.
· Risks and Actions have been reviewed.
Review Objectives Have Been Addressed
194
Current Status
· Reformatter toolkit SPSRB-required documentation development is currently in progress.
· All coding is complete for the Phase 1 SDR toolkit DAP. Unless any new issues arise, this is the version that will run operationally.
· The prototype DAP that NDE currently has should be able to run as soon as data are available after launch. N4RT developers will be available to provide any assistance and troubleshooting with the Toolkit for issues that may arise when data begin to flow.
195
Next Steps· After this review, the RID (Review Item Disposition) displaying the
project’s risks and actions will be updated. Any necessary edits to the Test Readiness Document will also be made.
· All modified project documents will be reposted to the project website and the updated links will be emailed to the reviewers.
· The Reformatter toolkit Algorithm Readiness Review will be held on 11/28/2011. The final review prior to delivery. It will be a working review to verify that the DAP is ready for implementation in the NDE environment and we’ll address and remaining risks.
· The Reformatter toolkit development team will deliver a final DAP to NDE on 11/30/2011. It will contain:» Final version of the code» Sample data» Final version of SMM, EUM, and TRR documentation. Note: there are sections in
the SPSRB documents that will be left blank as they are assigned to the “integrators” (NDE).
196
Open Discussion
· The review is now open for free discussion