Transcript
Page 1: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

Transition of Science Algorithms into Enterprise Product Generation Operations

2016 AMS Annual Meeting Dylan Powell, Ph.D.

Lockheed Martin ESPDS Science Engineer [email protected]

Rich Baker, Solers Inc. Peter MacHarrie, Solers Inc.

Ron Niemann, Solers Inc. Dan Beall, Solers Inc.

Page 2: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

2

2

Svalbard S-NPP, GCOM-W1

JPSS CGS IDPS

ESPDS NDE 1.0

Ingest Product Generation

Product Management SDRs & EDRs (xDRs)

Product Distribution: - S-NPP xDRs - NESDIS Unique Products (NUPs) - Tailored Products - GCOM-W L1/L2 Products

Ancillary Data

NASA VIIRS Image

ESPC

N

DE 1.0 Storage

Product and Ancillary Data Files

Product Distribution

ESPC Data Distribution

Services (DDS)

External/Internal Users Archive – CLASS (NUPs only)

Page 3: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

3

3

Svalbard/Fairbanks/ McMurdo

S-NPP, JPSS-1/2, GCOM-W1

JPSS CGS IDPS

NOAA Product Generation Services (NDE 2.0)

Ingest Product Generation

Product Management SDRs & EDRs (xDRs)

Products File Availability Notifications: - S-NPP/JPSS xDRs - NESDIS Unique Products (NUPs) - Tailored Products - GCOM-W L1/L2 Products

Ancillary Data File Availability Notifications

NASA VIIRS Image

ESPDS Common Infrastructure Platform

PDA (PD) Services

Product Distribution to Consumers Ancillary Data Acquisition

Enterprise Shared Storage

Product and Ancillary Data Files

Product and Ancillary Data Files

External/Internal Users Archive – CLASS (NUPs only)

Page 4: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

NOAA Data Exploitation (NDE) 1.0 in operations since July 2014 and has

been distributing critical S-NPP data since soon after launch (November 2011)

NDE 2.0 provides an Enterprise Product Generation (PG) framework implemented in a SOA within the ESPDS common infrastructure platform

Established and mature process for efficiently transitioning new and updated science algorithms from the NOAA Center for Satellites Applications and Research (STAR) - in place since 2011

Well defined, documented, and simple interface for “plug-and-play” algorithm integration allowing for a repeatable process and small integration team

Data agnostic, execute algorithms for any platform/instrument (“algorithm as a service” concept)

Established solution for product generation for S-NPP, JPSS-1/2, GCOM-W1, and can be leveraged for other NOAA or non-NOAA missions

4

Page 5: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

5

Ingest Dec 2014 Dec 2015

Ingested Data 103 TB 111 TB

Product Generation

Number of Production Jobs 1,510,997 1,591,465

Generated Data 25 TB 27 TB

Production Job Success Rate 99.992% 99.990%

Product Distribution

Distributed Data 29 TB 34 TB

Subscription Based Services

Number of Users 29 34

Number of Subscriptions (active)

254 375

Page 6: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

6 6

Develop Algorithm • Preliminary Design Review • Critical Design Review • Unit Test Readiness Review • System (Algorithm) Readiness Review

Prepare Delivered Algorithm Package (DAP) for each Science Algorithm for delivery to NDE • Algorithm Delivery Standards, Integration, and Test (DAP document acts as ICD) • Configuration management of DAP

Develop Algorithm Enhancements, Bug Fixes, and Updates • Redeliver DAP

Develop/test in NDE DEV Environment • NDE operating system/compilers • Science libraries • VPN access • Initial integration activities

Integrate DAP into NDE DEV Environment • Configuration management • Production rules • Unit testing

System Test Algorithm in NDE TEST Environment • Full contention, live data flow • Performance testing

Generate Operational Product in NDE Production Environment • Operations • 24/7 Monitoring

Satellite Products and Services Review Board (SPSRB) approves product for operations

STAR

NDE

KEY

1

2

3

5

6

7

Calibration/ Validation

Office of Satellite and Product Operations (OSPO) performs software code review 4

Define Requirements (JPSS Level 1)

8

Page 7: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

7

ESPDS Common Infrastructure Platform

NOAA Data Exploitation (NDE) PG

Enterprise Shared Storage

Science Algorithm - Single Driver Script - Read Process Control File (PCF) - Write output files to Process Status File (PSF)

Input Data

Science Algorithm - Log warnings and errors to master log file - Return code

PCF PSF Logs Return Code

Working Directory - Temporary directory for execution of production job - Dynamic input data - Write the PCF

Input/Output Data

PCF PSF Logs Return Code

Working Directory - Destroyed after successful execution - Failed jobs cause directory to be stored for later analysis

Input/Output Data

Page 8: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

Algorithm Platform/Primary Instrument

NDE 1.0 Operations Status

ESPDS PG (NDE 2.0) Baseline

Microwave Integrated Retrieval System (MiRS)

S-NPP/ATMS Operational 2014 QST-1

Advanced Clear-Sky Processor for Oceans (ACSPO)

S-NPP/VIIRS Operational 2014 QST-1

NOAA Unique CrIS/ATMS Processing System (NUCAPS)

S-NPP/CrIS Operational 2014 QST-1

VIIRS Polar Winds (VPW) S-NPP/VIIRS Operational 2014 QST-1

Green Vegetation Fraction (GVF) S-NPP/VIIRS Operational 2014 QST-1

S-NPP Tropical Cyclone S-NPP/ATMS Operational 2015 QST-1

Vegetation Health (VH) S-NPP/VIIRS Operational 2015 QST-1

GCOM-W1 AMSR2 Algorithm Software Processor (GAASP)

GCOM-W1/AMSR2 Operational 2015 QST-1

Active Fires (AF) S-NPP/VIIRS Operational 2016 QST-2

OMPS-Limb Profiler SDR S-NPP/OMPS-LP Operational 2016 ORR

8

Page 9: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

Algorithm Platform/Primary Instrument

NDE 1.0 Operations Status

ESPDS PG (NDE 2.0) Baseline

OMPS-Limb Profiler EDR S-NPP/OMPS-LP Operational 2016 ORR

JPSS Risk Reduction (Clouds, Aerosols, Cryosphere)

S-NPP, JPSS/VIIRS N/A QST-1

GCOM-W1 AMSR2 SDR GCOM-W1/AMSR2 N/A QST-1

9

Page 10: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

Microwave Integrated Retrieval System (MiRS)

Advanced Clear-Sky Processor for Oceans (ACSPO)

Sea Surface Temperature

Total Precipitable Water

Page 11: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

11

NOAA Unique CrIS/ATMS Processing System (NUCAPS)

VIIRS IR Image AWIPS2 NUCAPS Sounding

Page 12: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

12

Green Vegetation Fraction (GVF)

VIIRS Polar Winds (VPW)

Page 13: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

13

Vegetation Health (VH)

Tropical Cyclones (TC)

Page 14: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

14

Aggregation

Resample

NDE VIIRS Image

Filtering

Remapping

NDE VIIRS RGB NDE VIIRS RGB

NDE VIIRS RGB

375 m 3 km

NDE ACSPO SST

Page 15: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

15

Funny River Fire, Alaska VIIRS I1 (Visible)

VIIRS I3 (Fire)

Page 16: Transition of Science Algorithms into Enterprise Product ... - … · Transition of Science Algorithms into Enterprise Product Generation Operations . 2016 AMS Annual Meeting . Dylan

16

JPSS Risk Reduction Products

Clouds Snow/Ice Aerosols

GCOM AMSR2 Products

Winds

Clouds

Soil Moisture


Top Related