Climate Data Records

CDRs and CICS

J.J. Bates, J.L. Privette, E.J. Kearns, NOAANational Climatic Data Center

Asheville, NC

Outline

• NOAA Climate Goal reaction to NRC reports• Summary and status of NOAA’s CDR Program• Impact of CICS on CDR Program• Progress in JPSS-specific CDR issues and

strategic planning• F11 FFO

2

A Climate Data Record (CDR) is a time series of sufficient length, consistency, and continuity to determine climate variability and change FCDR: Calibrated signals for a family of sensors together with

the ancillary data used to calibrate them. (e.g. brightness temperatures, TOA radiances)

TCDR: Geophysical variables derived from FCDRs; May be generated by blending satellite observations, in-situ data, and model output (e.g., Aerosol Optical Depth, SST)

A Climate Information Record (CIR) is derived from CDRs and related data tand provides specific information about an environmental phenomena of importance to science and society (e.g., Hurricane tracks, Arctic Sea Ice Extent, Incidence of disease)

NOAA’s CDR Program is in response to an operational need

CDRs Provide Scientifically Corrected Long Time Series for Climate Analysis

Operational weather and hazard products are produced rapidly to potentially save life and property

Climate Data Records (CDRs)provide long term product consistency through rigorous reprocessing with advanced algorithms, ancillary data and evolved instrument understanding.

Climate Information Records (CIRs)provide specific information about environmental phenomena of particular importance to science and society (e.g., hurricane trends, drought patterns)

Vege

tatio

n G

reen

ness

Inde

x

Uncorrected Data Time Series Contain Both Environmental Information and

Satellite-induced Artifacts

New satellite

launched

Time (year)

Top of Atmosphere Vegetation Index

Top of Canopy Vegetation Index

Operational Climate vs

Operational Weather

5

CDR Program functionally located within RSAD at NCDC

NOAA NESDIS & Climate Goal reaction to NRC “Ensuring the Climate Record…”

• Clear Agency Roles and Responsibilities– Regular quarterly NOAA-NASA meetings and

NOAA participation in CERES and SORCE science team meetings are helping

• International Coordination– NOAA provides international leadership through

WMO GSICS & SCOPE-CM as well as CEOS Climate ad hoc Group

• Community Involvement in the Development of Climate Data Record

6

Management Board USGCRP Observations Management Structure

Product Development Teams (CDRs, CIRs and Support)

Engaging User Community

Modeling Monitoring Prediction Research

Experts in Instrument Characterization, Algorithms, Validation, Data

Management, Applications, and Observing System Performance Monitoring

Oth

er A

genc

ies

GE

OS

S, C

EO

SIn

tl D

ata

Pro

gram

s

Leve

ragi

ng R

esou

rces

/ C

olla

bora

tion

Observing System

Performance Monitoring

Production of Near Real-time

CDRs

Processing of CDRs for Long-term Records

Climate Information

Records

CDR Stewardship

Improved CDR and

CIRs

Design for Future

Systems

CDR Program Functional Framework

End-to-End Stewardship

8

END-TO-END STEWARDSHIP OF NOAA SATELLITE OBSERVATIONS

REQUIREMENT INPUT/OUTPUT QUALITY ASSURANCE ARCHIVAL OUTCOMES & IMPACTS

ESSENTIAL CLIMATE VARIABLE

OBSERVING SYSTEM OR

SENSOR

INSTRUMENT HEALTH

MONITORING

RAW DATA

RECORDS

ENVIRONMENTAL

DATA RECORDS

CALIBRATION &

VALIDATION

ARCHIVE STEWARD

SHIP

CLASS IT DEVELOP

MENT

CLASS OPERATIONS

CLIMATE DATA

RECORDS

CLIMATE SERVICES:

MONITORING UNDERSTANDING

MODLEING

Upper Air Temp, SST, Clouds, Sea

Ice… NPP / JPSSPARTY

RESPONSIBLE

ERB/TSI, Ozone

Climate Sensors (TSIS, CERES,

OMPS)

Cloud Properties GOES-R Sea Level JASON-3

Surface WindOSVW (Quickscat

replacement)

PrecipitationGCOM-W AMSR-

2 Ocean Color GCOM-C SGLI

Upper Air Temperature

COSMIC-2 / GPS-RO

NOTIONAL SAMPLE

CDR Activities Address 3 Epochs

1970 1980 1990 2000 2010 2020 2030

POES/GOES NPP JPSS/JASON-3/GOES-R

Establish operational CDRs and CIRs from NOAA’s archive of four decades of heritage data

Ensure climate quality data from operational systems, build Raw Climate Data Records to facilitate future

reprocessing, ensure continuity with heritage CDRs

Prepare for products from “Remanifested”

climate sensors and future missions

Long-Term Planning

• NOAA is initiating long-term planning by initiating community requirements meetings– Continuity of Earth Radiation Budget (CERB), July 2010

• NOAA is working with the international community to coordinate efforts– The aim of the WMO Sustained, Co-Ordinated Processing of

Environmental Satellite Data for Climate Monitoring (SCOPE-CM) is to establish a network of facilities ensuring continuous and sustained provision of high-quality satellite products related to the Essential Climate Variables (ECV), on a global scale, responding to the requirements of the Global Climate Observing system (GCOS).

– Committee on Earth Observing Satellites Climate Ad hoc Group (CEOS-CAG) - The group should ensure the more coherent product of FCDR/ECVs among agencies in response to the defined needs of GCOS

10

Preparing for CERES - ISCCP Reprocessing Data Flow

Multi-National Geostationary Satellite Data

Perform Mapping Operation

Run Cloud Detection Algorithm

Perform Radiative Retrievals

10 km pixel data

every 3 hours

Calculate Gridded Statistics 1 deg res

every 3 hours Merge the Various Satellites

Calculate Monthly Averages

B1

B4

BX

DX

DS

D1

D2 1 deg monthlyNCDC has

processed to this level to SW and LW Flux

calculations

CDR Program Funding Profile

Funding Mark or Source

Fiscal Year President’s Budget Request ($M)

Appropriated($M)

Redirected ($M)

ARRA($M)

2007 0 0 3.5 0

2008 0 0 0 0

2009 0 0 2.9 5.0

2010 7.0 10.0 0 0

2011 18.0 * * *

* To Be Determined

FY10 Investment Allocation*

13

Activity Investment (%)

Product Development 50

IT Systems 1

Research-to-Ops 5

Production and Stewardship

8

Science & CIRs 11

Mgmt./Planning/Outreach and Facility

26

100

*Expected**Anticipated

FY10 Organization Allocation*

14

Category Obligation (%)

Other NOAA (BOPs) 13

Other Federal Agencies 7

Universities: CICS-NC 23

Universities: Other CIs 14

Universities: Non-CI 11

Contracts 24

FTEs (NCDC) 6

100

13 7

231411

24 6Other NOAA (BOPs)Other Federal Agencies Universities: CICS-NC Universities: Other CIsUniversities: Non-CI Contracts FTEs (NCDC)

48%

* planned

NOAA NESDIS & Climate Goal reaction to NRC “Ensuring the Climate Record…”

• Microwave CDRs funded• Altimeter focus in FY11 FFO• Clouds and radiation (ISCCP)

reprocessing and R2O underway

• Hyperspectral being used to correct filtered radiometers

• Solar irradiance CDR funded• Global aerosol CDR R2O• Ocean color currently not

funded• Ozone CDR team funded

15

CDR Product Discipline Teams

• Goal– Have community experts coordinate activities within discipline/sensor– Develop coherent and self-consistent product suites– Ultimately seek consistency across disciplines (e.g., cloud detection)

• Possible Issues for Team to Address– Algorithms – adopt common cloud/land/water masks, FCDR and ancillary

data inputs, consistent QA flags and metadata for quality control, begin algorithm chain production and testing in development phase

– Products – identify gaps or resolve apparent redundancies within product suite; adopt common and appropriate grid projections, resolutions, data formats and packaging, etc.

16

Current CDR Development Portfolio(FY09 & FY10 Awards)

AVHRR (VIIRS)• Cloud Properties (Kato)• Snow/Ice (Key)• VNIR Cal./Clouds Minnis)• Thermal Calibration

(Mittaz)• Land/Carbon (Vermote)• Ocean Fluxes (Clayson)

SBUV (OMPS)• Ozone (Flynn)

DMSP: SSM/I, SSMIS (MIS)• Calibration (Kummerow)• Snow/Ice (Key)• Water Vapor (Luo)

SORCE, Glory (TSIS)• Solar Irrad.

(Pilewskie)

GOES: Imager (ABI)• VNIR Cal./Clouds (Minnis)

AMSU (ATMS)• Hydro Cycle (Ferraro)• Upper Air Temp (Ho)• Water Vapor (Luo) • Temp. Profile (Zou)

Other Satellites

Arrows identify key climate instruments

HIRS (CrIS)• FCDR/Intersensor calibration (Cao)• Water Vapor (Luo)• Cloud Properties (Menzel)

ERBS: ERBE (CERES)• Radiation Budget (Kato)

GPS RO (Various)• Temp. Profiles (Ho)

18

Coordination Activities:Annual CDR Team Meeting

• 2-day meeting (Aug. 4-5) in Asheville

• Attended by more than 50 participants from academia, industry, NOAA and other agencies

• Featured 14 competitively-selected CDR investigators describing algorithm and product plans

• CDR Program rolled out research-to-operations plans and expectations from team

• Impact: CDR investigators identified intra-team product dependencies & synergies; CDR Program developed detailed understanding of CDR algorithm formulation and maturity

19

TCDRsAtmospheric

Profiles Clouds & Aerosols PrecipitationOzone/Trace

Gases Oceans Land Cryosphere Solar Irradiance Radiation Budget

Discipline Lead Menzel Heidinger Ferraro Flynn Evans Vermote Key Pilewski Zhao

NCDC Tech. Lead Shi Knapp Nelson Shi Banzon Guillevic Zhang Zhao Zhao

NCDC IT Lead TBD TBD TBD TBD TBD TBD TBD TBD Hankins

Menzel (Shi)Heidinger (Knapp)

Boyles (Nelson) Flynn (Shi) Evans (Banzon)

Vermote (Guillevic) Key (Zhang) Pilewski (Zhao) Rossow (Zhao)

Ferraro (Kim) Minnis (Knapp) Ferraro (Kim) Long (Shi) Minnet (Zhang) Rossow (Zhao)Robinson (Guillevic) Lean (Zhao) Kato (Zhao)

Ho (Shi)Sarooshian

(Kim) Rosenlof (Shi)Reynolds (Banzon) Meier (Zhang) Pap (Zhao) Lee (Shi)

Luo

(Knapp/Young) Casey (Banzon) Lazlo (Zhao)

Zou (Shi) Clayson (Zhang) Li (Zhao)

Mears (Nelson) Heidinger (Zhao)

FCDRs Imagers (SW/IR)Imagers (u-

wave) Sounders (TIR)Sounders (u-

wave)Sounders (Trace

Gas)Solar

Radiometers ERB Instruments GPS-RO

Team Leader Evans Kummerow Cao Ferraro Flynn Pilewski Kato Ho

NCDC Tech. Lead Knapp Semenegus Shi Kim Shi Zhao Zhao Shi

NCDC IT Lead TBD TBD TBD TBD TBD TBD TBD TBD

Evans (Banzon)Kummerow

(Semenegus) Cao (Shi) Ferraro (Kim) Flynn (Shi) Pilewski (Zhao) Kato (Zhao) Ho (Shi)

Mittaz (Knapp)Wentz

(Semenegus) Menzel (Shi) Luo (Nelson) Rice (Zhao) Rossow (Zhao)

Vermote

(Guillevic) Lee (Shi) Zou (Shi) Lean (Zhao)

Minnis (Knapp)

Heidinger (Knapp)

New CDR Teaming at NCDC

Public Posting of Prototype CDRs in Operations

20

• Maturity Matrix adopted by GCOS (1 of 12 guidelines for CDR Production)

CDRs in Development

21

Online Data• Investigator Teams/

Affiliations• Abstracts• Progress Reports• Community Workshop URLs• Team Meeting Presentations

Impacts• Used by EUMETSAT’s

Satellite Application Facility on Climate Monitoring (CM-SAF)• Interest in coordinating work

CDR Program Management: Staffing

• Federal Program Manager vacancy announcement open– FAC-P/PM Senior certification– Closes 9/10

• RFP for Program Management Support released by ERAD– 2 year base + 1 year option– 4 Key Personnel positions– Technical review completed Aug. 20– Expected award date: this week?

22

Interaction with CICS

• Several selected CDR grants were awarded as Type III funding

• Support for key internal work– NEXRAD, extreme events, SST

• Securing science and scientific programmer support in RSAD• Postdocs and students• Assembling IT Development Environment• Summer ‘11 climate workshop

Impact of Interaction with CICS

• Introduction of academic flavor to an operational federal workplace

• Part of paradigm shift at NCDC• Enabling CDR processing

– Development environment• Network connections• Bring another dimension to NCS

JPSS-Specific Issues

• NASA missions are end-to-end and NOAA are broken into segments– NOAA has mapped one to the other and

found gaps that are being addressed• Sensor integration and testing• Instrument scientist and analyst• Sensor cal/val teams• CDR legacy processing not portable

25

CDR Information Flow

Sensor DataRecords (SDRs)

Data (Direct & Remotely Sensed- RDRs)

Fundamental Climate Data

Records (FCDRs)

Thematic Climate Data Records (TCDRs)

Homogenization and Calibration

Time-tagged Geo-Referenced

EnvironmentalData Records

(EDRs)

Converted to Bio-Geophysical

Variables

Climate Information Records (CIRs)

Climate Data Records or Homogenized Time Series

Fundamental Climate Data Record (FCDR): Time series of calibrated signals for a family of sensors together with the ancillary data used to calibrate them.

Thematic Climate Data Record (TCDR): Geophysical variables derived from FCDRs, often generated by blending satellite observations, in-situ data, and model output.

Raw Climate Data Records (R-CDRs)

Recent AccomplishmentsMajor Milestones

Project Schedule

• System Definition Review (Completed: Mar. 2010)

• Critical Design Review (Scheduled: Sept. 16, 2010)• “R-CDR CDR” • External reviewers from GSFC & Univs. Miami,

Wisconsin & Colorado

Updated: 03/12/10

Project Risks• Complexity of the NPP/NPOESS RDRs and short

schedule• Staffing needs for ATMS, and OMPS NP Raw CDRs

and system execution coding

• Hired 2 new programmers; 1 more sought

• Successfully ported and adapting NPP/VIIRS and CrIS software as used by NASA PEATES

R-CDR

Interface Data

Processing Segment

Land PEATE

NPP Operational Software Reuse

Climate Raw Data Record(R-CDR) Status

NPP/JPSS GSFC NCDC

FY11 FFO

• http://www.federalregister.gov/a/2010-17294/p-579 • Initially focused on Fundamental CDRs and Thematic CDRs related to

Earth's water and energy cycles and sea level • Funding Availability: For the present grants competition, the CDR Program

expects to select proposals over a two year period (FY 2011 and FY 2012) for funding. The total anticipated Federal funding in FY 2011 is $2.5M for new awards. The anticipated number of new awards is from 5 to 15.

• The CDR Program anticipates new funding availability in FY 2012 for additional awards from the present grants competition.

• The total anticipated Federal funding in FY 2012 and the number of additional awards will be dependent on the enacted budget.

• Application Deadline: Letters of Intent (LOI) should be received at the CDRP Office no later than 5 p.m. Eastern Time, September 15, 2010

• Full proposals must be received no later than 5 p.m. Eastern Time, November 10, 2010

Summary

• The CDR Program has recurring resources and is actively engaging the U.S. and international climate communities to operationalize CDRs for NOAA.

• CICS has already proven to be a key cooperator for the CDR Program

• JPSS gaps in climate information stewardship have been identified and are being addressed

• FY11 FFO is out -- and responses due soon!

29

Questions?

NPP Data Flow Diagram

NESDISCentral

AFWACentral

NPP/NPOESS Data ProcessingSegment (IDPS)

Archive & Dist Segment

(CLASS; NCDC and NGDC)

Ingest

RoboticStorageDatastore

Inventory

Science Data

Segment(NASA)

Cal/Val Program

ERB CARS

NDE(NPOESS

Data Exploitation)

PEATEs

Cal.

Production &

Validation

STAR/GSICS

DataProcessing

&Re-Processing

Char.

CDR

CIR

RDRs

xDRs, IPs, Ancillary& Auxiliary Data

SDRs, EDRs, IPs,

Ancillary &

Auxiliary Data

Users

CERES CDRs

NUPs

KEY:RDR Raw Data RecordSDR Sensor Data RecordTDR Temperature Data Record EDR Environmental Data RecordxDR RDR, SDR, TDR, EDRCDR Climate Data RecordCIR Climate Information RecordIP Intermediate ProductNUP NOAA Unique Product

xDRs, IPs

V1.6 NCDC NPP 16 July 2009

RDRs

CDRs,CIRs

Research Communit

yOMPS-Limb SDRs, EDRs

SDRs, TDRs,

IPs, EDRs

xDRs, CDRs, NUPs

CERES CDRs

PEATEsPEATEsPEATEs

Correlative

Data &

Results

CDR Program (NCDC)

CIRs

, IPs

, Anc

illar

y&

Aux

iliar

y Da

ta

xDRs, IPs, Ancillary& Auxiliary Data

RDRs, Ancillary& Auxiliary Data

R-CDRs

Climate Raw Data Record (C-RDR)System Interfaces

xDRs, IPs, Ancillary& Auxiliary Data

CDR/CIR Processing

CLASS NCDC R-CDR

Processing

Archive

National Climatic Data Center (NCDC)

NESDIS CentralInterface

Data Processin

g Segment

(IDPS) CLASS PoP

Climate User Community

Outline

• About NOAA’s new CDR Program at the National Climatic Data Center

• Why NOAA is interested in this activity for oceans, land, & atmosphere

• What is meant by an operational CDR• How it will engage the science

community to meet program goals

FY10 CDR Investments

34

Area Description Primary Vehicle(s)

Development – Products Algorithms, Prototype Products, Docs

Grants, CI, IAAs

Development – IT System Computing, Networks Contracts

Research-to-Operations Code conversion, optimization, IV&V

CI, Contracts

Production & Stewardship

(Re-)Processing O&M, Archive, Distribution

Contracts

Science and CIRs CDR analysis and characterization CI

Mgmt/Planning/Outreach and Facility

Incl. Project docs, Summer Institute, User Reqts. Group, Renovation

Contracts, CI

Maturity Matrix Identifies Milestones and Research-to-Operations Transition Points

Sustained, Coordinated Processing of Environmental Satellite Data for Climate Monitoring (SCOPE-CM)

The aim of SCOPE-CM is to • establish a network of facilities ensuring continuous and sustained provision

of high-quality satellite products related to the Essential Climate Variables (ECVs)

• global scale• respond to the requirements of the Global Climate Observing system

(GCOS)