ghrc data processes · data maturity model recommendation 10: develop a data maturity model for...

Presented at the GHRC User Working Group Meeting October 7, 2015

GHRC DATA PROCESSES Lifecycle, Levels of Service, Maturity Model Helen Conover GHRC Operations Manager [email protected]

GHRC Dataset Lifecycle Formalized GHRC dataset management processes in Lifecycle and Levels of Service documents •  Reviewed lifecycle documents from NOAA and multiple

DAACs (NSIDC, PO.DAAC, LP DAAC)

•  Reviewed GHRC practices and procedures

•  Assessed GHRC on Peng’s stewardship maturity matrix for digital environmental data

https://ghrc.nsstc.nasa.gov/home/ghrc-docs/data-management

10/7/2015 2 User Working Group Meeting

Peng, G., Privette, J. L., Kearns, E. J., Ritchey, N. A., & Ansari, S.. (2015). A Unified Framework for Measuring Stewardship Practices Applied to Digital Environmental Datasets. Data Science Journal, 13(0), 231–253. DOI: http://doi.org/10.2481/dsj.14-049

New Dataset Evaluation


DAAC Process for Implementing New Data Types and/or Services (As Is)

3.0

ES

DIS

Pro

ject

2.0

DA

AC

1.0

DA

AC

Use

r Wor

king

G

roup

4.0

NA

SA

HQ

Ear

th S

cien

ce D

ata

Sys

tem

E

xecu

tive

Identify request for supporting

new data type or service

2.2ESDIS Review

Required

Implement New Data Type or

Service

Review Holdings, Product

Templates, and Impact

Assessments

Complete Product Templates and

Impact Assessments

Review New Request

3.1NASA HQ

review required?

Generate Rejection

Justification

Review/Update Rejection

Justification

3.2Approve Request?

Review New Request

4.0Approve

Request?


Justification

1.2Modify

Request?

Return to Start

End

StartYes

No

Yes

No

Yes

No

No

Yes

Yes

No

2.1UWG review

required?

1.1Recommend

Implementation?

Yes

No

Archival Interest Form

DAAC appropriate

?

Email DP with appropriate alternate archives

DP

DC

Data Provider

Dataset Coordinator

DP

DC

Dataset Ingest Process Planning Ingest Documenta<on Publica<on

Answer Data

Provider Ques<ons

Upload Sample data

Confirm Submission

Collect ini<al metadata

Assign soBware developer

Verify Data Set

completeness

Publish Data Set

Monitor submission

Ini<ate data set

submission

Send ini<al email to DP

Verify data file names and loca<ons

Assign Documenta<on Coordinator

Create/Edit Metadata

Review landing page and guide doc

Provide documents

Ingest / archive

dataset and documents

Configure ingest/archive

soBware

Rename / reformat scripts (if needed)

Data Provider

Dataset Coordinator

SoBware Developer

Documenta<on Lead

DP

DC

SW

DL

Outreach OR

News items: • Weekly notes • Social media • GHRC web site

Op<onal: • Earthdata feature • Email announcement

DP

DC

SW

DL

OR

DP

DC

SW

DL

OR

Thanks to ORNL DAAC to swimlanes graphic

New Dataset Versions Planning Ingest Documenta<on Publica<on

No<fy DAAC that

new version is available

Confirm Submission

Update ini<al metadata

Assign soBware developer

Verify Data Set

completeness

Publish Data Set

Configure ingest/archive

soBware

Verify data file names and loca<ons


Provide updated

documents

Ingest / archive

dataset and documents

Review / update rename /

reformat scripts (if needed)

Answer Data

Provider Ques<ons for new version

Create/Edit new version Metadata Review both

landing pages and guide

docs Update previous

version metadata to reference new

version

Con<nue to Re(re Dataset

Re<re previous version?

News items: • Weekly notes • Social media • GHRC web site

Op<onal: • Earthdata feature • Email announcement

DP

DC

SW

DL

OR

DP

DC

SW

DL

OR

Data Provider

Dataset Coordinator

SoBware Developer

Documenta<on Lead

DP

DC

SW

DL

Outreach OR

Retire a Dataset

Op<ons to re<re a dataset: ① Leave data available online with low level of service ② Remove data from online server and public catalog,

keep on archive ③ Remove from online server, catalog and archive ④ Transi<on to long term archive

Request to re<re a dataset

Prepare Data Assessment Package

Re<re?

Request to re<re a dataset

Remove Data Set


Sta<c landing page, reference new version if applicable

No more metadata or

service updates

Re<re Request Evalua<on Documenta<on Re<re Dataset

Package data, metadata and

docs

1

Transi<on Data Set

DP

DC

UWG

DL

ESDIS Re<re?

4 2

3

NASA ESDIS Project

Data Provider

Dataset Coordinator

DP

DC

User Working Group UWG

Documenta<on Lead DL

ESDIS

DP

DC

UWG

DL

ESDIS

1 2 3 4

DAAC Process for Implementing New Data Types and/or Services (As Is)

3.0

ESD

IS P

roje

ct2.

0D

AAC

1.0

DAA

C U

ser W

orki

ng

Gro

up

4.0

NAS

A H

QEa

rth S

cien

ce D

ata

Syst

em

Exec

utiv

e

Identify request for supporting

new data type or service

2.2ESDIS Review

Required

Implement New Data Type or

Service

Review Holdings, Product

Templates, and Impact

Assessments

Complete Product Templates and

Impact Assessments

Review New Request

3.1NASA HQ

review required?

Generate Rejection

Justification


Justification

3.2Approve Request?

Review New Request

4.0Approve

Request?


Justification

1.2Modify

Request?

Return to Start

End

StartYes

No

Yes

No

Yes

No

No

Yes

Yes

No

2.1UWG review

required?

1.1Recommend

Implementation?

Yes

No

ESDIS-‐UWG review process

Levels of Service

Data collections at the GHRC DAAC may be handled with different levels of service (LoS). •  For some aspects of data services, such as ingest

method, LoS corresponds to characteristics of the data. •  For other aspects of data services, LoS will depend on

overall data handling priority assigned to the general categories of GHRC data holdings

CATEGORIES*OF*DATA*SERVICES*

Off/site*Backup* Data*Ingest*Post/Ingest*Processing*

Metadata*and*Documentation*

Distribution*Services*

Cloud,'other'DAAC'

Automated,'ongoing'

Product'generation' Guide'document' Exploration,'

analytics'Tape'copy' Periodic'ingest' Reformat' README' Visualization'PI'institution' Bulk'download' Rename' DOI'and'citation' Access'services'' PI'upload' None' Catalog' FTP/HTTPS'

Dataset Priorities Priority' '''''''''''''''''''''''''''''''''GHRC'DATA'CATEGORIES''

SATELLITE'MISSIONS'1" NASA"satellite"datasets"(OTD,"TRMM"LIS,"ISS"LIS,"AMSU)"1" Airborne"validation"datasets"(LIP,"multiple"campaigns)"2" Ground"validation"datasets"–"open"access"(LMA)"3" Other"satellite"datasets"(DMSP"OLS,"NOAA"MSU)"5" Ground"validation"datasets"–"commercial,"restricted"access"

(Vaisala/NLDN,"WWLLN,"ENGLN)"MEaSUREs'PROGRAM'

1" DISCOVER"(RSS)"FIELD'CAMPAIGNS'and'EARTH'VENTURES'(Hurricane'Science'or'GPMAGV)'1" NASA"research"instruments"(airborne"or"ground,"NASANsponsored"PI)"2" Affiliated"research"instruments"(e.g.,"from"partner"university)"3" Other"agency"research"instruments"(e.g.,"sponsored"by"NOAA,"DOE)"4" Ancillary"research"data"(e.g.,"PERSIANN,"TRMM"flood"maps)"5" Other"agency"operational"data"(e.g.,"GOES"imagery,"NWS"radar)"

NASA'APPLICATIONS'Research'Results'1" Applications"products"(e.g.,"SANDS"analysis"products)"3" Selected"input"products"(e.g.,"MODIS"subsets"for"selected"storms)"

Data Maturity Model Recommendation 10: Develop a data maturity model for GHRC data. Provide this on website and include maturity information for each dataset provided. Review NOAA’s data maturity model as a starting point. •  Also looked at NASA’s data maturity levels

o  Beta – gain familiarity with data parameters and formats o  Provisional – initial data exploration and process studies o  Validated Stage 1 – selected independent measurements o  Validated Stage 2 – peer reviewed literature o  Validated Stage 3 – quantified uncertainty o  Validated Stage 4 – systematic validation updates


NOAA: http://www1.ncdc.noaa.gov/pub/data/sds/maturity-table-6level.pdf NASA: http://science.nasa.gov/earth-science/earth-science-data/data-maturity-levels/

THANK YOU for your attention Questions? Please contact GHRC User Services for any help or questions [email protected]


ghrc data processes · data maturity model recommendation 10: develop a data maturity model for...

Documents