OSTP Phase II Assessment: Climate Record Generation

Briefing toPanel on Options to Ensure the Climate Record from the

NPOESS and GOES-R Spacecraft

Jeffrey L. PrivetteNOAA’s National Climatic Data Center

19 June 2007

2

Outline

• Background• NOAA’s Climate Data Record Framework• OSTP Climate Record Generation Assessment• Next Steps

3

• National Academy has emphasized the need for Climate Data Records

– Current and past NASA and NOAA efforts have been ad hoc, need-driven and experimental

• NOAA initiated the Scientific Data Stewardship project to develop and implement a long-term, systematic,and comprehensive approach

– Rooted in NRC dialog

– Planning and project is joint with NASA

Climate Data Record Generation Background

© 2004Other key NRC reports:

•Global Environmental Change: Research Pathways for the Next Decade (1999)

•Ensuring the Climate Record from the NPP and NPOESS Meteorological Satellites (2000)

•Climate Change Science: An Analysis of Some Key Questions (2001)

•Planning Climate and Global Change Research (2003)

4

NRC Identified Key Elements of a Successful CDR Program

CDR Organizational Elements

• High-level leadership council

• Advisory council to represent climate research community and other stakeholders

• Fundamental Climate Data Record (FCDR) Teams

• Thematic Climate Data Record (TCDR) Teams

CDR Organizational Elements

• High-level leadership council

• Advisory council to represent climate research community and other stakeholders

• Fundamental Climate Data Record (FCDR) Teams

• Thematic Climate Data Record (TCDR) Teams

CDR Generation Elements

• High accuracy and stability of FCDRs

• Pre-launch characterization of sensors and lifetime monitoring

• Thorough calibration of sensors

• Well-defined criteria for TCDR selection

• Stakeholder involvement and feedback for TCDRs

• Well-defined criteria for TCDR validation

• Use of in-situ data for validation

CDR Generation Elements

• High accuracy and stability of FCDRs

• Pre-launch characterization of sensors and lifetime monitoring

• Thorough calibration of sensors

• Well-defined criteria for TCDR selection

• Stakeholder involvement and feedback for TCDRs

• Well-defined criteria for TCDR validation

• Use of in-situ data for validation

Sustaining CDR Elements

• Available resources for reprocessing CDRs as new information becomes available

• Provisions for feedback from scientific community

• Long-term commitment of resources for generation and archiving of CDRs and associated data

Sustaining CDR Elements

• Available resources for reprocessing CDRs as new information becomes available

• Provisions for feedback from scientific community

• Long-term commitment of resources for generation and archiving of CDRs and associated data

Fundamental Climate Data Record (FCDR): Time series of calibrated signals for a family of sensors together with the ancillary data used to calibrate them.

Thematic Climate Data Record (TCDR): Geophysical variables derived from FCDRs, often generated by blending satellite observations, in-situ data, and model output.

CDR - A time series of measurements of sufficient length, consistency, and continuity to determine climate variability and change

5

Observational RequirementsWeather vs. Climate

• Weather– Best instant snapshot of an

individual product– Weather variations are large

compared to time-dependent observation biases

– Always marches forward using the present to predict the near future

• Climate– Best blend of all snapshots from

a family of records– Climate variations are small

compared to time-dependent observation biases

– Revisits the past to provide a richer context for the present and future

6

Sensor DataRecords (SDRs)

Data (Direct & Remotely Sensed)

Fundamental Climate Data

Records (FCDRs)

Thematic Climate Data Records

(TCDRs)

Climate Data Records or Homogenized Time Series

Homogenization and Calibration

Time-tagged Geo-Referenced

Converted to Bio-Geophysical

Variables

EnvironmentalData Records

(EDRs)

Converted to Bio-Geophysical

Variables

What is Needed for EDRs vs. CDRs?

7

Sensor DataRecords (SDRs)

Data (Direct & Remotely Sensed)

Fundamental Climate Data

Records (FCDRs)

Thematic Climate Data Records

(TCDRs)

Climate Data Records or Homogenized Time Series

Homogenization and Calibration

Time-tagged Geo-Referenced

Converted to Bio-Geophysical

Variables

EnvironmentalData Records

(EDRs)

Converted to Bio-Geophysical

VariablesMature

Developing

What is Needed for EDRs vs. CDRs?

8 Prelaunch

Characterization

Peak EDR knowledge(applied to final products only)

CDRs Exploit AccumulatedSensor and Algorithm Knowledge

Sen

sor

Kn

ow

led

ge

(~ P

rod

uct

Qu

alit

y)

Time (years)

Launch

Checkout &Characterization

End of Operations

Operational Lifetime

Peak CDR knowledge(applied to complete record)

Long-termClimate

Mission Duration

9

AVHRR Surface Reflectance EDR Shows Spurious Jumps and False Trends

Result: Incorrect Climate Conclusions

Problem: Orbital drift caused change in observation time EDR depicts erratic trends and variability due to observatory rather than environment.Solution: Reprocess with calibration and orbital corrections CDR provides proper record of environmental variability.

Error Source

11

End-to-End CDR Planning Landscape:Prioritization, Production, & Knowledge

Knowledge

Global Essential ClimateVariables with Heritage Records

* With Certified NPOESS Program (Post Nunn-McCurdy)

*

13

Example Key Uncertainties, Gaps in Knowledge, and Research Needs Identified in

the IPCC 4th Assessment Report

• Changes in Human and Natural Drivers of Climate• Solar changes - TSIS• Modification of clouds by aerosols – VIIRS, APS, ERBE• Radiative forcing due to stratospheric water vapor – CrIS• Atmospheric methane – CrIS• Tropospheric ozone – CrIS• Land surface properties – VIIRS

Production Example - Sea Surface Temperature

Linkages & processes needed to create daily SST product:

Climate-quality Algorithm Complexity Far Exceeds That for Weather

Higher Quality, Known Uncertainties, Connection to heritage record

15

Climate Product Hierarchy Designed to Address IPCC4 Key

Questions

ECVsECVs

Climate Data RecordsClimate Data Records

Climate Information ProductClimate Information Product

SurfaceWinds,

Cloud Properties

Ocean Vector Winds,Cloud Top Temperatures

Trends in Atlantic Hurricane IntensityTrends in Atlantic Hurricane Intensity

IPCC4: Limitations on hurricane frequency, intensity, and data interpretation

Major Climate Uncertainties

Product Taxonomy Example

16

Phase II Assessment andBackground

• Phase I Assessment (January 2007): “Impacts of NPOESS Nunn-McCurdy Certification on Joint NASA-NOAA Climate Goals”

• Phase II Study: “Identifying Alternatives in Response to Joint NASA-NOAA Climate Goals”

• Terms of Reference (21 February 2007)

– Joint NASA-NOAA Study

– NASA Primary Role• Analyze and cost options for space measurement restoration

– NOAA Primary Role• Analyze and cost processes to (re-)produce, archive, and

steward climate quality data records

17

Phase II Climate Product Considerations

• Key Issues

1. Climate Product Generation is an essential element for development/continuity/stewardship of global climate record

2. Priorities are to minimize measurement gaps and provide climate products

3. Starting point: assume raw instrument data are available in archive• Data may come from non-NPOESS programs, incl. foreign partners• Address all Certified / Demanifested / Alternative instruments• CDRs merge current and relevant heritage satellite and in-situ data

4. Lifecycle Cost Estimates: Through FY26 (19 years)

5. Roles and Responsibilities currently under discussion

18

ECVs Can Require Data Fusion

• ALBEDO (SURFACE)• LAND SURFACE TEMP• LAND COVER• OCEAN COLOR• VEGETATION LEAF AREA• VEGETATION FAPAR• FIRES DISTURBANCE

• CLOUD PROPERTIES• SEA ICE• SNOW COVER• SEA SURFACE TEMP• UPPER AIR WIND (POLES)

• SOLAR IRRADIANCE

• AEROSOL PROPERTIES

• PRECIPITATION•SURFACE WIND SPD & DIRECTION

• ATMOS. WATER VAPOR• UPPER AIR TEMP

• OZONE• METHANE, CO2, GHG

• RADIATION BUDGET (SFC & TOA)

• SEA STATE• SEA LEVEL

VIIRS(14)

TSIS(1)

APS(1)

MIS (and OVW CAPABILITY)(10)

CrIS/ATMS(4)

OMPS(2)

ERBS/CERES)(1)

Altimetry(4)

• LAKES

• GLACIERS/ICE CAPS

Plus derived NOAA-Unique Information Products: e.g., Hurricane trends, trends in extreme temperatures and rainfall

Sensor

Essential Climate

Variables (#)Climate Data Records

(#) Subproducts* (#)

VIIRS 11 32 180

CrIMMS 3 14 105

MIS 5 8 76

OMPS (Nadir Sensor Only) 1 4 5

CERES (C1 Platform Only) 1 2 12

TSIS 1 2 6

Altimeter 2 2 16

APS 1 3 30

Totals: 25 67 430

*Subproducts include data in different assemblages, resolutions, formats, etc.

De

ma

nif

es

ted

Pa

rtia

lly

Ce

rtif

ied

Ce

rtif

ied

ECV, CDR and Product Countsby NPOESS Sensor

20

Key CDR Activities

Climate Algorithm Development and Refinement (Leveraged)

Characterization, Time-varying Bias Adjustment, Seamless Records (Leveraged)

Data (Re-)Processing (New for climate)

Archiving and Distribution (New for climate)

Data Stewardship (New for climate)

CDR Science and Information Product Development (New for Climate)

21 21

Climate Record Generation Key Activity Relationships

Data Stewardship(Ensures interoperability

of subsystems, integrity

of complete system,

and long-term data preservation

and extendability)

Instrument and Data

Characterization

Algorithm Development,

Refinement & Validation

Data (Re-)Processing

Archiving &

Distribution

CDR Science &

Information Product

Development

Joint agency development to provide long-term, systematic and comprehensive CDR generation

22 22

What Is Data Stewardship?

• Ensures interoperability of all CDR subsystems, integrity of complete system, and long-term data preservation and extendability

• Begins pre-launch (e.g., guidance and co-development with system segments)

• Required throughout lifecycle (e.g., CM, documentation, tools, product support)

• Examples of Pre-launch and Early Mission Stewardship Requirements– Develop Policies and Processes to Ensure Traceability, Provenance and Reproducibility

• Metadata– Developing system-level standards– Sufficient for cataloging and user access– Developing product-by-product information and formats (pixel, granule, and collection levels)

• Auxilliary, Ancillary and Intermediate Product Archiving and Distribution– Develop CDR Configuration Control System

• Versioning, Filename structures, Data Retention/Deletion Process– Document and Configuration Manage: Algorithms, Subsystems, and Products

• Algorithm Theoretical Basis Documents, Algorithm Design Docs, Detailed Design Docs, Hardware Description– Develop CDR Uncertainty Definition and Reporting Process

• e.g., MODIS Stage 1, 2, 3 validation definitions– Develop Characterization/Cal/Val data Ingest/Archive/Distribution Framework

• Metadata, Documentation and Configuration Control• Intermediate Product archiving/distribution requirements

– Develop Guidelines for CLASS Product Assembling: Aggregation, Bundling, Compositing

23

Hypothetical Transitioning of Agency Roles & Responsibilities

1 2 3 4 5 6 7 8 9 10 11 12

Year (Relative)

Algorithm Development/Refinement

Characterization, Trend Adjustment

Data (Re-)Processing

Archiving and Distribution

Data Stewardship

Science, Applic. & Info. Prod. Dev.

NOAA

NASA

24

Next Steps

• Listen closely to discussions/recommendations from this Workshop

• Refine CDR Science Support budget and risk estimation

• Continue to work with OSTP

• Continue dialogue regarding potential international and/or domestic partnerships

Backups

26

Total ECVs & CDRs

MISMIS

27

• Societal benefits of the Global Earth Observations System of Systems (GEOSS) emphasize the tradition of applied climatology but using 21st century systems

• Key Science Intergovernmental Panel on Climate Change (IPCC) Issues address the policy-relevant scientific research topics of highest priority for Climate and Global Change studies

Prioritizing – Key Uncertainties, Gaps in Knowledge, and Research Needs Identified in

the IPCC 4th Assessment Report

28

Production “Climate Central” Concept

– For reprocessing, NOAA’s Scientific Data Stewardship (SDS) requires an IDPS-like system (Climate Central) to process:

– SDS interdependent with

CLASS, e.g., large data I/O– Reprocessing– Requires full production

engineering

NOAA/CLASS Users

ACQUIRE RDRs

IMPROVED SDRsMULTISATELLITE

FCDRs

PROCESS TCDRs

PROVIDEASSESSMENTSIMPROVEMENTS

NOAASDS

CLIMATECENTRAL

Products and Services

Provide FCDRs and TCDRs

RDR SDR Multi-satellite FCDRs & TCDRs

29

Assumptions for “Cost to Nation” Scenario

• Climate Record Generation– All CCSP/GCOS satellite-based ECVs plus NOAA-unique ECVs (e.g.,

hurricane metrics)– NASA budgeting used for demanifested sensors

• Traditional sensor and science teams• Includes funding of field validation networks/campaigns

– NOAA budgeting used for Certified sensors• Team size linked to CDRs assigned to sensor• Validation based on operational field networks (e.g., buoys, sondes,

Climate Ref. Net)– NOAA Stewardship Augmentation

• Climate Central activities beyond traditional Science Team responsibilities– Agency roles and responsibilities not yet defined

• Research-to-Operations transition expected

30

Science SupportBudget Development

Cost

ECVCov

erag

e

Cost/Schedule Reliability

■ Cost_to_Nation (complete) ■ Detailed_Model (Summer 2007)

Approach:1. CDR-based budgeting spreadsheet

provided Rough Order of Magnitude (ROM) Budget for May deliverable

2. Comprehensive “Bottom-Up” cost model in development

31

Characterization: Inventory of TasksExample

Pre Launch• Design Review/Analy./Accept.• Trade Study Support

–EDR Impact Analyses

• On-site T/V Testing Support • EDU Subsystem, SystemTest Data Analysis• FU Subsystem, System Test Data Analysis• System/Algorithm Coefficient Table Generation

–ISO Reference Traceability

• Operational Data Monitoring System Design& Build• In-situ Cal/Val Network Gap Analysis/Augmentation

–Protocols for C/V Data Collection, Metadata, Archiving and Distribution

• Inter-satellite Cal/Val System Design/Build

–Acquisition Coordination

Launch & Checkout• Diagnostic Telemetry Analy. • Onboard Cal System Checkout• System/Algorithm Coefficient Table Revision (on-orbit data)• Sensor Diagnostic Operation and Data Analysis• Calibration Field Campaigns• Inter-satellite Calibration Evaluation• Lunar Calibration Analysis (if applicable)

Operational Life• Diagnostic Telemetry Analy.• Cal Data Monitoring/Trending

–Intra-orbit Seasonal Analy. –Long Term Stability Analy.

• Cal/Val Field Campaigns• Cal/Val Network Operations and Maintenance• SDR/EDR Product Validation• System/Algorithm Coefficient Table Updates and Tuning• Anomaly Detection, Analysis & Resolution• Graceful Degradation Support• Algorithm Modifications to Accommodate Degradation• s/c Maneuver Support and Analysis (if applicable)

Long-term Climate• Heritage Data Calibration/Rectification

–Inter-satellite–Vicarious–Lunar–Develop Dynamic System Coefficient Tables

• Multisatellite Long Term Stability Analysis and Correction• CDR Product Validation and Error Characterization• Heritage Data Anomaly Detection, Analysis & Resolution• Heritage System Graceful Degradation Development• Algorithm Modifications to Accommodate Heritage System Anomalies/Artifacts (e.g., orbital drift)

32

Next Steps:Roles and Responsibilities

Space Is Complex

Activity

Time

Contributors

Considerations

• Activity set (coarse scale) is readily developed

• Contributor set must be developed, evaluated• Agency, Interagency, Industry, Academia

• Timing must address multiple challenging factors• “Research-to-Operations” hand-offs• Existing/Funded Projects and Capabilities

• Possible extended EOS / POES lifetimes• Proposed Projects and Capabilities• Mission roadmap decisions• NRA responses (proposals) difficult to predict

33

Proposed Path Forward Defining Tasks, Roles & Responsibilities

1. Develop task inventory for each major Climate Support activity Tasks, subtasks may vary with activity, sensor and/or product

2. Determine agencies/organizations with heritage capabilities

3. Determine projects that can cover needs with existing funds e.g., NPOESS operational program, NASA SDS, NOAA STAR/GSICS

4. Identify gaps in current capabilities and funding

5. Develop plan for allocating uncovered tasks among agencies Time varying Justify overlaps where needed

6. Develop plan for allocating uncovered tasks to organizations within agency Justify overlaps where needed

7. Roll up into master budget profile

34

Team Organization

• Steering: Mary Kicza and Mike Freilich• Leads:

– NOAA: Jeff Privette and Jim O’Neal– NASA: Bryant Cramer and Steve Neeck– IPO’s PEO/SUAG Representative: Mike Bonadonna

• Ex Officio (NOAA): Dave Young, Mike Tanner, Chet Koblinsky, Tom Karl

• Working Group (NOAA):– NOAA Chief Financial Office: Darrell Robertson and Mahendra Shrestha– Cost Modeling

• Consultants: John Bates (NOAA) and Bruce Barkstrom (NOAA)– Sensor Expert Teams

• TSIS: Rossow (City University of New York [CUNY]), Barkstrom (NOAA)• ERBS: Rossow (CUNY), Ackerman (Wisconsin)• ALT: Wilson (NOAA), Miller (NOAA)• OMPS: Flynn (NOAA), Ravishankara (NOAA)• APS: Murphy (NOAA), Laszlo (NOAA)• CMIS: Janowiak (NOAA), Chang (NOAA), Chelton (Oregon), Arkin (Maryland)• CrIMMS: Barnett (NOAA), Goldberg (NOAA)• VIIRS: Menzel (Wisconsin), Reynolds (NOAA), Privette (NOAA)• Support: Goldberg (NOAA), Menzel (Wisconsin), Justice (UMD), Weng (JCSDA)