npoess preparatory project validation program for the imagery … · npoess preparatory project...

1
NPOESS Preparatory Project Validation Program for the Imagery and Cloud Mask Data Products from the Visible/Infared Imager/Radiometer Suite Thomas Kopp 1 , Richard Frey 2 , Keith Hutchison 3 , and Barbara Iisager 3 , with H. Kilcoyne 4 and A. Griffin 5 1 The Aerospace Corporation, Omaha, NE 2 NOAA/NESIDS/STAR/CIMMS, Madison, WI 3 Northrop Grumman Aerospace Systems, El Segundo, CA 4 NOAA/NESDIS/STAR, Greenbelt, MD 5 Praxis INC, Alexandria, VA Overview: The Joint Polar Satellite System (JPSS) Program will launch the NPOESS Preparatory Project (NPP), a risk reduction and data continuity mission, prior to the first operational JPSS launch. The NOAA/NESDIS Center for Satellite Applications and Research (STAR)-led Data Products and Algorithms (DPA) Group of the JPSS Ground Project will execute the NPP Validation program in collaboration with subject matter experts from the user communities to ensure the data products comply with the requirements of the sponsoring agencies. The Imagery and VIIRS Cloud Mask (VCM) plans incorporate a combination of real-time users, scientists, and climate specialists to insure a viable and effective set of products. The Imagery team includes subject matter experts and organizations representative of all aspects of imagery application, from clouds to sea ice to aerosols along with ground/ocean features of meteorological interest. The VCM plan considers not only the VCM as a single product, but extensive evaluation regarding its impacts on all downstream VIIRS products. Both leakage and false alarms are accounted for, and users dependent upon the confidently cloudy and confidently clear output are included as part of the evaluation team. Evaluation tools include the capability to display each cloud detection test, match ups between the VCM with CloudSat, CALIPSO, and MODIS, quantitative comparisons between the VCM and manual analysis produced by subject matter experts, and statistical impacts on dependent Environmental Data Products such as SST and surface reflectance. This comprehensive approach will allow all dependent users to obtain a reliable measure of the capabilities of the Imagery and VCM. This presentation will highlight pre-launch activities and provide an overview of the approaches, data and schedule for the validation of the NPP VIIRS Imagery and Cloud Mask data products. Imagery Validation Team Name Organization Thomas Kopp Aerospace Pablo Clemente-Colon NIC Chris Elvidge NOAA/SNCDC Jeff Hawkins NRL-Monterey Keith Hutchison NGAS Mike Plonski NGAS Kim Richardson NAVO Jeff Tesmer FNMOC Develop/Adapt tools for EDR Assessment. Evaluate quality of EDR, emphasizing assessment of interferences with usefulness, such as striping, scattering, ghosting, etc., evaluate EDR based on operational viability to Customers and quantify performance against System Specification. Imagery is heavily employed by many customers. As such, LTM will be covered by feedback from the operational users. The SDR Cal/Val plan covers LTM for the VIIRS SDRs, and most items that would impact imagery would be captured by their efforts. PRE-LAUNCH EARLY ORBIT CHECK-OUT (EOC) INTENSIVE CAL/VAL (ICV) LONG-TERM MONITORING (LTM) Analyze the initial EDRs for issues and coordinate with SDR calibration team. Imagery Environmental Data Record (EDR) Approach: Evaluating imagery involves both Subject Matter Experts (SME) and non-experts, and considers the diverse uses of imagery across the spectrum. It includes critical interaction with the VIIRS Sensor Data Record (SDR) Cal/Val team, as most of the potential problems with imagery would be addressed by the SDR experts. Our approach first screens the usefulness of the imagery by SME; then submits the imagery for assessment by the remaining team members and their associated organizations. Data Product Maturity Definitions Beta The imagery EDR is produced consistently but has undergone only minimal validation and is not suitable for operational use. Provisional The imagery has passed step 1 above for evaluation by the Centrals, but has not been accepted by any of the Centrals as operationally viable. Validated Stage 1 The Centrals acknowledge that for clouds and sea ice, the two atmospheric items that the System Specification has explicit requirements to meet, imagery is suitable for operational use. Validated Stage 2 The Collaborative Team agrees the imagery meets or beats the System Specification. Validated Stage 3 The Centrals acknowledge that the imagery meets all operational viability criteria, to include real-time applications not considered in the System Specification. These include identification of SST gradients, use in aerosol detection, and NexSat applications. Work will be in two sequential “steps” both during the Intensive Cal/Val (ICV) phase. SMEs will evaluate the quality of the imagery itself. This small group will be comprised of three SMEs, and will focus on the overall quality of the imagery with an emphasis on items that would interfere with the usefulness of the image, to include striping, scattering, ghosting, and errors in band-to-band registration (BBR) that prevents proper overlays of the imagery. Customer SMEs will evaluate imagery based on their needs and the “operational viability” criteria that defines’ their use of imagery, and the Collaborative Team will perform the quantitative analysis of the imagery relative to the System Specification. VIIRS Cloud Mask Validation Team Name Organization Thomas Kopp Aerospace John Eylander AFWA Richard Frey NOAA/CIMSS Jeff Hawkins NRL-Monterey Andrew Heidinger NOAA/CIMSS Keith Hutchison NGAS Barbara Iisager NGAS Doug May NRL-Stennis Jeff Tesmer FNMOC Eric Vermote UMCP/GSFC VIIRS Cloud Mask Internal Data Product Approach: Evaluating VCM involves Subject Matter Experts (SME) in cloud detection algorithms and imagery interpretation, with significant coordination with SMEs for each of the VIIRS EDR products. The VCM Cal/Val effort will basically be two steps: 1. 30-day rapid tuning event to produce a VCM adequate for other EDR products to employ. This step commences when the SDR and Imagery is determined adequate for VCM evaluation and is intended to allow for validation of the dependent downstream EDRs to begin earlier in ICV 2. Simultaneous set of studies performed to fully characterize the VCM’s performance. These studies are expected to take approximately 6-9 months after the initial 30-day tuning process. a) quantitative analysis using manual determination of cloud over a specified set of VIIRS granules. b) quantitative comparison of other cloud algorithms with the VCM. c) quantitative analysis comparing the VCM to other external sources of cloud measurements such as CALIPSO and CloudSat. Other sources such as LIDAR and ARM sites also apply, though their benefits are limited for the VCM. Comparisons to cloud masks created from MODIS and AVHRR are planned. d) incorporating feedback from the other Cal/Val disciplines and EDRs, especially SST and surface reflectance, into tuning corrections that maximize the quality of the VCM. SST effects will be analyzed by NAVO, who will run the SST algorithm using different cloud masks and focus on specific regions to isolate cloud types causing issues and detecting anomalies. Land EDR effects will be studied using a time series of surface reflectance to detect adverse affects from potential errors in the VCM and will report errors to the VCM team. Developed a tool for quantitative VCM assessment against System Specification, tuned algorithm using global synthetic data, evaluated tuning against MODIS proxy data, and analyzed algorithm condition sensitivities using PEATE processed operational code and correlative data match-ups. Complete coordination with the other VIIRS EDR Cal/Val teams maximizing the quality of the VCM output for each of the 20+ EDRs dependant on the VCM. Finalize the long term approach to tuning the VCM for the various customers and users. Document the results. Consider additional cloud detection tests or alterations as suggested by either the Cal/Val team members or the Algorithm Working Group from STAR. Perform 30-day rapid tuning for use with downstream algorithms once SDRs and Imagery products are adequate for use and characterize VCM performance via a quantitative analysis using manual determination of cloud over a specified set of VIIRS granules, a quantitative comparison of other cloud algorithms with the VCM, a quantitative analysis comparing the VCM to other external sources of cloud measurements such as CALIPSO, and incorporating feedback from the other Cal/Val disciplines and EDRs, especially SST and surface reflectance, into tuning corrections that maximize the quality of the VCM. Transition applicable tools and techniques to Operations and Sustainment team for monitoring product performance Monitor product performance and correct when needed. Monitor sensor performance and analyze impact on product performance. INTENSIVE CAL/VAL (ICV) LONG-TERM MONITORING (LTM) A-TRAIN 2010 Above are two examples of comparisons between the VCM and MODIS cloud masks using CALIOP as truth. For this example, the VCM is untuned (i.e. thresholds are not optimized) and the algorithm is using MODIS data as a proxy for VIIRS. The results span one year’s worth of comparisons, with the daytime results to the left and nighttime results to the right. Even in an untuned state, the VCM is generally competitive with the MODIS cloud mask, and within 10% for most cases. Recent pre-launch work after tuning indicates the daytime analysis is approaching MODIS quality. The table at the right shows VCM performance before and after tuning by land cover type against the NPOESS system specification (which will be replaced by a JPSS system specification). PRE-LAUNCH EARLY ORBIT CHECK-OUT (EOC) Untuned VCM Land Ocean Desert Snow PCPC SysSpec 38.3 15 19.2 15 14.7 15 58.0 15 False Alarms Sys Spec 12.4 7.0 6.0 5.0 10.5 N/A 22.5 N/A Leakage SysSpec 0.5 3.0 0.3 1.0 2.3 N/A 0.02 N/A Prob. Correct Typing SysSpec 87.1 90.0 93.7 94.0 87.2 N/A 77.4 N/A Tuned VCM PCPC SysSpec 7.5 15 22.5 15 3.5 15 5.0 15 False Alarms Sys Spec 4.6 7.0 1.4 5.0 2.1 N/A 6.2 N/A Leakage SysSpec 1.4 3.0 0.6 1.0 3.7 N/A 0.6 N/A Prob. Correct Typing SysSpec 93.4 90.0 97.5 94.0 93.9 N/A 92.8 N/A

Upload: others

Post on 25-Jun-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: NPOESS Preparatory Project Validation Program for the Imagery … · NPOESS Preparatory Project Validation Program for the Imagery and Cloud Mask Data Products from the Visible/Infared

NPOESS Preparatory Project Validation Program for the Imagery and Cloud

Mask Data Products from the Visible/Infared Imager/Radiometer SuiteThomas Kopp1, Richard Frey2, Keith Hutchison3, and Barbara Iisager3, with H. Kilcoyne4 and A. Griffin5

1The Aerospace Corporation, Omaha, NE 2NOAA/NESIDS/STAR/CIMMS, Madison, WI 3 Northrop Grumman Aerospace Systems, El Segundo, CA 4NOAA/NESDIS/STAR, Greenbelt, MD 5Praxis INC, Alexandria, VA

Overview:

The Joint Polar Satellite System (JPSS) Program will launch the NPOESS Preparatory Project (NPP), a risk reduction and data continuity mission, prior to the first operational JPSS launch. The NOAA/NESDIS Center for Satellite Applications and Research (STAR)-led Data Products and Algorithms (DPA) Group of the JPSS Ground Project will execute the NPP Validation program in collaboration with subject matter experts from the user communities to ensure the data products comply with the requirements of the sponsoring agencies. The Imagery and VIIRS Cloud Mask (VCM) plans incorporate a combination of real-time users, scientists, and climate specialists to insure a viable and effective set of products. The Imagery team includes subject matter experts and organizations representative of all aspects of imagery application, from clouds to sea ice to aerosols along with ground/ocean features of meteorological interest. The VCM plan considers not only the VCM as a single product, but extensive evaluation regarding its impacts on all downstream VIIRS products. Both leakage and false alarms are accounted for, and users dependent upon the confidently cloudy and confidently clear output are included as part of the evaluation team. Evaluation tools include the capability to display each cloud detection test, match ups between the VCM with CloudSat, CALIPSO, and MODIS, quantitative comparisons between the VCM and manual analysis produced by subject matter experts, and statistical impacts on dependent Environmental Data Products such as SST and surface reflectance. This comprehensive approach will allow all dependent users to obtain a reliable measure of the capabilities of the Imagery and VCM. This presentation will highlight pre-launch activities and provide an overview of the approaches, data and schedule for the validation of the NPP VIIRS Imagery and Cloud Mask data products.

Imagery Validation Team

Name Organization

Thomas Kopp Aerospace

Pablo Clemente-Colon NIC

Chris Elvidge NOAA/SNCDC

Jeff Hawkins NRL-Monterey

Keith Hutchison NGAS

Mike Plonski NGAS

Kim Richardson NAVO

Jeff Tesmer FNMOC

Develop/Adapt tools for EDR Assessment.

Evaluate quality of EDR, emphasizing assessment of interferences with usefulness, such as striping, scattering, ghosting, etc., evaluate EDR based on operational viability to Customers and quantify performance against System Specification.

Imagery is heavily employed by many customers. As such, LTM will be covered by feedback from the operational users. The SDR Cal/Val plan covers LTM for the VIIRS SDRs, and most items that would impact imagery would be captured by their efforts.

PRE-LAUNCH EARLY ORBIT CHECK-OUT (EOC) INTENSIVE CAL/VAL (ICV) LONG-TERM MONITORING (LTM)

Analyze the initial EDRs for issues and coordinate with SDR calibration team.

Imagery Environmental Data Record (EDR) Approach:

• Evaluating imagery involves both Subject Matter Experts (SME) and non-experts, and considers the diverse uses of imagery across the spectrum. It includes critical interaction with the VIIRS Sensor Data Record (SDR) Cal/Val team, as most of the potential problems with imagery would be addressed by the SDR experts. Our approach first screens the usefulness of the imagery by SME; then submits the imagery for assessment by the remaining team members and their associated organizations.

• Data Product Maturity Definitions– Beta – The imagery EDR is produced consistently but has undergone only minimal

validation and is not suitable for operational use. – Provisional – The imagery has passed step 1 above for evaluation by the Centrals, but

has not been accepted by any of the Centrals as operationally viable.– Validated Stage 1 – The Centrals acknowledge that for clouds and sea ice, the two

atmospheric items that the System Specification has explicit requirements to meet, imagery is suitable for operational use.

– Validated Stage 2 – The Collaborative Team agrees the imagery meets or beats the System Specification.

– Validated Stage 3 – The Centrals acknowledge that the imagery meets all operational viability criteria, to include real-time applications not considered in the System Specification. These include identification of SST gradients, use in aerosol detection, and NexSat applications.

• Work will be in two sequential “steps” both during the Intensive Cal/Val (ICV) phase.– SMEs will evaluate the quality of the imagery itself. This small group will be comprised

of three SMEs, and will focus on the overall quality of the imagery with an emphasis on items that would interfere with the usefulness of the image, to include striping, scattering, ghosting, and errors in band-to-band registration (BBR) that prevents proper overlays of the imagery.

– Customer SMEs will evaluate imagery based on their needs and the “operational viability” criteria that defines’ their use of imagery, and the Collaborative Team will perform the quantitative analysis of the imagery relative to the System Specification.

VIIRS Cloud Mask

Validation Team

Name Organization

Thomas Kopp Aerospace

John Eylander AFWA

Richard Frey NOAA/CIMSS

Jeff Hawkins NRL-Monterey

Andrew Heidinger NOAA/CIMSS

Keith Hutchison NGAS

Barbara Iisager NGAS

Doug May NRL-Stennis

Jeff Tesmer FNMOC

Eric Vermote UMCP/GSFC

VIIRS Cloud Mask Internal Data Product Approach:

Evaluating VCM involves Subject Matter Experts (SME) in cloud detection algorithms and imagery

interpretation, with significant coordination with SMEs for each of the VIIRS EDR products. The VCM

Cal/Val effort will basically be two steps:

1. 30-day rapid tuning event to produce a VCM adequate for other EDR products to employ. This step

commences when the SDR and Imagery is determined adequate for VCM evaluation and is intended

to allow for validation of the dependent downstream EDRs to begin earlier in ICV

2. Simultaneous set of studies performed to fully characterize the VCM’s performance. These studies are

expected to take approximately 6-9 months after the initial 30-day tuning process.

a) quantitative analysis using manual determination of cloud over a specified set of VIIRS granules.

b) quantitative comparison of other cloud algorithms with the VCM.

c) quantitative analysis comparing the VCM to other external sources of cloud measurements such

as CALIPSO and CloudSat. Other sources such as LIDAR and ARM sites also apply, though their

benefits are limited for the VCM. Comparisons to cloud masks created from MODIS and AVHRR

are planned.

d) incorporating feedback from the other Cal/Val disciplines and EDRs, especially SST and surface

reflectance, into tuning corrections that maximize the quality of the VCM.

SST effects will be analyzed by NAVO, who will run the SST algorithm using different cloud masks

and focus on specific regions to isolate cloud types causing issues and detecting anomalies.

Land EDR effects will be studied using a time series of surface reflectance to detect adverse

affects from potential errors in the VCM and will report errors to the VCM team.

Developed a tool for quantitative VCM assessment against System Specification, tuned algorithm using global synthetic data, evaluated tuning against MODIS proxy data, and analyzed algorithm condition sensitivities using PEATE processed operational code and correlative data match-ups.

Complete coordination with the other VIIRS EDR Cal/Val teams maximizing the quality of the VCM output for each of the 20+ EDRs dependant on the VCM. Finalize the long term approach to tuning the VCM for the various customers and users. Document the results. Consider additional cloud detection tests or alterations as suggested by either the Cal/Val team members or the Algorithm Working Group from STAR.

Perform 30-day rapid tuning for use with downstream algorithms once SDRs and Imagery products are adequate for use and characterize VCM performance via a quantitative analysis using manual determination of cloud over a specified set of VIIRS granules, a quantitative comparison of other cloud algorithms with the VCM, a quantitative analysis comparing the VCM to other external sources of cloud measurements such as CALIPSO, and incorporating feedback from the other Cal/Val disciplines and EDRs, especially SST and surface reflectance, into tuning corrections that maximize the quality of the VCM.

• Transition applicable tools and techniques to Operations and Sustainment team for monitoring product performance

• Monitor product performance and correct when needed.

• Monitor sensor performance and analyze impact on product performance.

INTENSIVE CAL/VAL (ICV) LONG-TERM MONITORING (LTM)

A-T

RA

IN 2

01

0

Above are two examples of comparisons between the VCM and MODIS cloud masks using CALIOP as truth. For this example, the VCM is untuned (i.e. thresholds are not optimized) and the algorithm is using MODIS data as a proxy for VIIRS. The results span one year’s worth of comparisons, with the daytime results to the left and nighttime results to the right. Even in an untuned state, the VCM is generally competitive with the MODIS cloud mask, and within 10% for most cases. Recent pre-launch work after tuning indicates the daytime analysis is approaching MODIS quality. The table at the right shows VCM performance before and after tuning by land cover type against the NPOESS system specification (which will be replaced by a JPSS system specification).

PRE-LAUNCH EARLY ORBIT CHECK-OUT (EOC)

Untuned VCM Land Ocean Desert Snow

PCPC

SysSpec

38.3

15

19.2

15

14.7

15

58.0

15

False Alarms

Sys Spec

12.4

7.0

6.0

5.0

10.5

N/A

22.5

N/A

Leakage

SysSpec

0.5

3.0

0.3

1.0

2.3

N/A

0.02

N/A

Prob. Correct Typing

SysSpec

87.1

90.0

93.7

94.0

87.2

N/A

77.4

N/A

Tuned VCM

PCPC

SysSpec

7.5

15

22.5

15

3.5

15

5.0

15

False Alarms

Sys Spec

4.6

7.0

1.4

5.0

2.1

N/A

6.2

N/A

Leakage

SysSpec

1.4

3.0

0.6

1.0

3.7

N/A

0.6

N/A

Prob. Correct Typing

SysSpec

93.4

90.0

97.5

94.0

93.9

N/A

92.8

N/A