npoess data exploitation (nde) preliminary design review november 21, 2006 nsof building 4231...

NPOESS Data Exploitation (NDE)Preliminary Design Review

November 21, 2006NSOF Building 4231 Suitland Road Room 2001

Welcome

-J. Silva

3

PDR Agenda 8:00 a.m. Welcome

8:15 a.m. Introduction/Background

9:00 a.m. Project Overview

10:45 a.m. NDE Context and Hardware

11:15 a.m. Algorithm Development

12:00 p.m. Lunch

1:00 p.m. Data Handling System

2:30 p.m. IT Security

3:00 p.m. Communications Study

3:40 p.m. Closing Remarks

4:00 p.m. PDR Board Review of RFAs

4

Agenda

8:00 a.m. Welcome





12:00 p.m. Lunch






• Introduction/Background J. Silva

• Mission & Design Drivers J. Silva

• Architectural Overview G. Goodrum

• Data Products T. Schott

5

NDE Design Review Board

– Chairperson: Vanessa Griffin, (NESDIS/OSD) Ground Systems Division Chief

– Mike Mignogno (NESDIS/OSD) Polar Program Manager– Mike Tanner (NESDIS/HQ) Senior Program Advisor – Reginald Lawrence (NESDIS/OSDPD), Environmental Satellite Processing

Center Chief– David Benner (NESDIS/OSDPD) Satellite Services Division Chief– Mitch Goldberg (NESDIS/STAR) Satellite Meteorology and Climatology

Division Chief– Joseph Mulligan (NOAA/IPO) NPOESS Interface Data Processing Segment

Lead – Kevin Schrab, Ph.D. (NWS) Satellite Product Improvement Manager– Brian Gockel (NWS/OST/SEC) Systems Engineer– Lt. Amanda Bittinger (NESDIS/STAR) Satellite Oceanography and

Climatology Division – Charles MacFarland NESDIS/OCIO Representative

6

Request for Action

Briefing Title: Slide Number: _________________________________________ Comment/ Question:

_________________________________________ Background on Comment or Question:

_________________________________________ Team response:

___________________________________________Submitted by: _________

Briefing Title: Slide Number: _________________________________________ Comment/ Question:

_________________________________________ Background on Comment or Question:

_________________________________________ Team response:

___________________________________________Submitted by: _________

• RFAs are available on tables at back of room• RFAs to be submitted to a Review Board Member• All RFAs are due today by the end of the meeting

(4:00 p.m.)– Please state your questions clearly– Review Board will disposition RFAs

• Board Members are requested to convene to disposition RFAs

• RFAs are available on tables at back of room• RFAs to be submitted to a Review Board Member• All RFAs are due today by the end of the meeting

(4:00 p.m.)– Please state your questions clearly– Review Board will disposition RFAs

• Board Members are requested to convene to disposition RFAs

7

The NDE IT Design Team

Jim SilvaGary SloanTom Bishop

Geof GoodrumRaj KhannaGary Roth

Tom Schott

Management Team

Geof GoodrumStan Cutler

Lou FenichelRaj Khanna

Jim SilvaEd Wilson

Requirements

Emily GrahamESPC CM Team

ConfigurationManagement

Barbara HickmanRich Mumaw

Tom SuhrstedtCustomer Reps

Comm. Study

Tom BishopBrian Callicott

Wendy HolmesBarbara Hickman

ESPC Architecture

Maurice McHughTony Conrad

OSDPD Product LeadsBill Reupke

STAR Product LeadsKrishna Tewari

Algorithm I/F Design

Geof GoodrumRaytheon

Peter MacHarrieEd Wilson

IDPS I/F Design

Ann Al-JazrawiStan Cutler

Geof GoodrumBarbara Hickman

Raj KhannaPeter MacHarrie

Gary RothHongming Qi

System Modeling

Brian MerandiPeter MacHarrieAnn Al-Jazrawi

CLASS I/F Design

8

Today’s Objective

• Objectives: To ensure that NDE’s design concepts are consistent with NPOESS and NOAA architectures and that NDE plans are feasible

• NDE Design Review Board roles and responsibilities: – Review NDE system conceptual design – Review project plans necessary to reach Critical Design Review

(CDR) in August 2007– Review strategies (acquisition, development, testing,

implementation, transition) – Assess risks– Request actions that will improve NDE designs and plans

9

Assessment Criteria

• Does the proposed preliminary design satisfy the contractual requirements?

• Has the overall system framework been clearly presented? • Has the functionality been appropriately allocated to

subsystems?• Have the interfaces between subsystems and to external

entities, including users, been identified?• Have the security features and controls been defined?• Have delivery strategies been defined for the subsystems (build,

buy, reuse)?• Are the plans and schedules leading to CDR feasible?

10

Background Documents

• NDE Requirements & Performance Measures

– Software Engineering

– Infrastructure

– Systems Management

– Data Retention & Archive

– Interfaces

– Operational Product Generation

– Communications

& Distribution

• NDE Requirements & Performance Measures

– Software Engineering

– Infrastructure

– Systems Management

– Data Retention & Archive

– Interfaces

– Operational Product Generation

– Communications

& Distribution

See NDE Web site: http://projects.osd.noaa.gov/nde

• Project Plan v.2– Scope

– Organizational Views

– Roles and Responsibilities

– Project Approach

– Schedule

– Acquisition Strategy

– Budget

• Project Plan v.2– Scope

– Organizational Views

– Roles and Responsibilities

– Project Approach

– Schedule

– Acquisition Strategy

– Budget• Concept of Operations

– Scope

– Justification

– System Concepts

– Operational Scenarios

– Impacts

– Analysis of Proposed System

http://projects.osd.noaa.gov/nde

http://projects.osd.noaa.gov/nde

11

NDE Design Review Board:

Review Calendar

Date Deliverable TaskedNov 21 Requests for

Action (RFA)Review Board

Dec 15 Action Responses to

Board and Chairperson

NDE

Design Team

Jan 26 Final Report Design Board Chairperson

12

Agenda

8:00 a.m. Welcome





12:00 p.m. Lunch






• Introduction/Background J. Silva




13

Mission

The NPOESS Data Exploitation (NDE) will provide critical environmental products

derived from NPP and NPOESS observations to NOAA’s operational

community in near-real time

14

Capabilities Assessment:NDE’s Δs

Current StateCurrent State Target StateTarget State

DMSP + POESDMSP + POES ΔΔ NPP + NPOESSNPP + NPOESS

End-to-End ControlEnd-to-End Control ΔΔ Post-processingPost-processing

Orbital ProcessingOrbital Processing ΔΔ Granule ProcessingGranule Processing

Stove-pipe Applications Stove-pipe Applications ProcessingProcessing ΔΔ Centralized Data ProcessingCentralized Data Processing

Bolted-on SecurityBolted-on Security ΔΔ Security Compliant w/ StandardsSecurity Compliant w/ Standards

100 GB/day Ingest100 GB/day Ingest ΔΔ ~2.5 (NPP) TB/day Ingest ~2.5 (NPP) TB/day Ingest

300 GB/day Distribution300 GB/day Distribution ΔΔ ~6 TB/day Distribution~6 TB/day Distribution

Heritage InstrumentsHeritage Instruments ΔΔ New Instrument TechnologiesNew Instrument Technologies

15

• Granule (data structure) Processing

• Instrument specific • SafetyNet

• “Continuous” ingest • Near-real time = <30 minutes

• All data packaged as HDF5

NDE Project:Capabilities Assessment

Current StateCurrent State Target StateTarget State

Orbital ProcessingOrbital Processing ΔΔ Granule ProcessingGranule Processing

• Orbital Processing • Near-real time = 156 minutes from observation to product

NDE systems must adapt to all new input formats, alter formats to user needs, take

advantage of improved latency

16

• Centralized IT Architecture• Reusable objects • Database Technologies & Data Management• Common CM, testing, and transition processes• Common toolsets• Centralized management and control


Current StateCurrent State Target StateTarget State Stove-pipe Stove-pipe Applications Processing Applications Processing ΔΔ

• Stove-pipe Applications Processing

• Duplicated, redundant functionality• High cost of management & control

NDE will establish latest proven technologies and manage them in 3 basic environments: Development, System Test, Operations

Centralized IT ArchitectureCentralized IT Architecture

17

• Increase customer base (increase benefits)

• Online subscriptions• GUIs

• More Dissemination Options:•Push, pull, network, fiber, internet, etc.


Current StateCurrent State Target StateTarget State 300 GB/day Distribution 300 GB/day Distribution ΔΔ ~6 (tbc) TB/day Distribution~6 (tbc) TB/day Distribution

• Product benefits realized by relatively small customer base

• Limited number of dissemination options

• “orders” require developer intervention• FTP (pull) in most cases

NDE to establish a Communications Architecture in collaboration with NOAA’s Network Advisory

Committee (NAC) and AWIPS

18

• Environmental satellite industry challenge– NESDIS history: ~ 2 ½ years from data availability, through

research, to provision of products to operational users (R2O)

• All new NOAA projects must have Transition Teams and submit Transition Plans for approval

– NDE submitted plans (MIRS, CrIS/ATMS, Sea Surface Temperature) developed with SPSRB oversight

• NDE’s Goals– Minimize IT obstacles as algorithms migrate through

development, test, and operational environments– Work with end users to prepare operations for products

The Transition Challenge

19

NDE Milestone Plan

20

Agenda• Introduction/Background J. Silva




8:00 a.m. Welcome





12:00 p.m. Lunch






21

NDE System Objectives

• Disseminate NPOESS Data Records to NOAA Operational Community

• Generate and disseminate tailored NPOESS Data Records (versions of NPOESS Data Records in previously agreed alternative formats and views)

• Generate and disseminate NOAA-unique products (augmented environmental products constructed from NPOESS Data Records)

• Deliver NOAA-unique products, product processing elements, and associated metadata to CLASS for long-term archiving

• Provide services to customers, including NDE product training, product enhancement, and implementation support across NOAA

• Provide software for NPOESS Data Record format translation and other data manipulations

22

NPOESS Data Exploitation (NDE) Mission Context Diagram

NPOESSGroundSystem

NPOESSGroundSystem

NOAA Environmental Satellite Processing Center(ESPC)

NOAA UniqueProducts

TailoredProducts

TailoringToolsData Records

(xDRs)

NOAA UniqueProducts

NPOESS willdeliver more than100 different datarecords (xDRs)

Data Records (xDRs)

Data Records(xDRs)

Near Real-time Customers



ClimateCommunity

NOAA-Unique Products

Tailoring Tools

xDRs CLASS

NPOESSData

Exploitation(NDE)

CLASSCLASS

Near Real-timeCustomers

Near Real-timeCustomersClimate

Community

ClimateCommunity

23

Scope & Requirements

• Design, develop and implement– Development Environments for

NPP/NPOESS– System Test Environment for NPP/NPOESS– Operational Environment for NPOESS

NDE Phasing

Requirements & Preliminary Planning

PreliminaryDesign

Critical Design

Release Release

Release Release

Closeout

Deployment Lifecycle

24

Vision:3 Environments, Common Standards

Environmental Satellite Processing Center

Shared Capabilities, Common Standards

NDEOperationalEnvironment

NDESystem

Test/BackupEnvironment

NDEDevelopmentEnvironment

25

Design Considerations

• Algorithm modularity

• Reuse

• Centralization

• Scalability

26

IDPS / NPP & NDE Integration Timeline

…

SWIC

Qual liteComplete

2/28/07

B1.4:

QUAL-Lite

CUT Complete

4/21/061.4 CDW11/14/05

SWICComplete

7/7/06

SegInt

Continuous Segment INT

ACI Design

ACI ATP 8/30/05 ACI CDW 2/23/06

ACI CUT

CUT

Seg IntComplete11/10/06

Performance:

IDPS Algorithm Sci2Ops

Performance Analysis & Prototyping

Note: New NPP IPAC Verification events (Performance Verification) not yet Planned

IDPS Algorithm Sci2Ops

Detailed Design

SWICComplete

7/27/07

QUALSegInt

Pre-FATInt

4/11/08

Cont Segment INT

CDW1/11/07

CUTComplete

5/30/07

B1.5: CUTBAR Prep

BAR10/12/06

Alg FODeadline3/30/07

SWIC

Seg IntComplete10/30/07

FAT

Int

QualComplete

4/4/08

Initial HW Procure 10/1/07

Site HW Procure / Pre-Ship

1/4/08

FAT Install and Prep

Concepts in work

2Q20072Q2005 3Q2005 4Q2005 1Q2006 2Q2006 3Q2006 4Q2006 1Q2007 3Q2007 4Q2007 1Q2008 2Q2008 3Q2008 1Q20094Q20082Q20072Q2005 3Q2005 4Q2005 1Q2006 2Q2006 3Q2006 4Q2006 1Q2007 3Q2007 4Q2007 1Q2008 2Q2008

Small algorithm changes to SPCR work-off

NPP Launch9/30/09

NPP Sys Test / Sys Readiness Activities

FATComplete

6/27/08

FAT

NESDIS SAT/

AFWA SIT

9/30/08

A-SATN-SAT

Site Installs and Prep

Late/large algorithm changes

O&S

Characterization Updates/Final Verif

Final PerfVerif

3/30/09(TBR)

NDE-NPPIntegration

NDE System TestEnvironment

NDE CriticalDesign

NDE PreliminaryDesign

27

NDEPreliminary

Risks

Stakeholders Inputs

RiskDatabase

FY05 OMB 300

Risks

RiskSurvey

Analysis of Conse-

quences

Likelihood & Impact

NDE RiskManagement

Plan and Assessment

NDE Objectives

Prioritized Risks

Identify Risk Handling & Mitigation

Risks & Risk Handling

Risk Handling and MitigationAssessment of impact and project plans

• Mitigating steps in NDE’s current plans– Develop a risk management plan– For IT work, schedule control via earned value

management– Develop lifecycle cost and benefit estimates– Develop the project budget– Plan for needed resources– Design systems to support reliability requirements– Monitor NOAA and DoC security requirements

• New risk mitigating activities– Develop and implement a quality management plan

to address system and data quality, quality assurance, and quality control.

– Develop and implement NDE continuity plans that are coordinated among participating organizations.

– Coordinate development of preliminary (pre-launch) data with the IPO and instrument designers

28

Risk Management Summary

• Risks management plan will be revised as risks are handled, new risks are identified, or stakeholder views change– Continued input from

stakeholders and other sources

– Developers have a critical responsibility to identify issues, risks, and challenges and their potential impact on operations

• Preliminary design review will identify issues and challenges for key design elements

29

Agenda• Introduction/Background J. Silva




8:00 a.m. Welcome





12:00 p.m. Lunch






30

Product Definitions• Products delivered by NPOESS contractor

– xDRs• Raw Data Records (RDR)• Sensor Data Records (SDR)• Temperature Data Records (TDR)• Environmental Data Records (EDR)

– Intermediate Products (IP)– Pre-planned Product Improvements (P3I)

• Products produced by NOAA using xDR information– Tailored xDRs and IPs – NOAA Unique Products (NUPs)

31

NPOESS Data Exploitation (NDE)

Integrated Program Office (IPO)

NPOESS Products Requirements

NPOESS Integrated Operational Requirements

Document (IORD – II)Dec 10, 2001

NPOESS Technical Requirements Document

Sensor, Temperature,

Environmental Data Records

(xDRs)

Requirements Document Contract Document Deliverables

NDE Product Matrix (1) Tailored xDRs (2) NOAA-unique products (NUPs)

NDE Product Development

Current Polar

Products

NOAA Mission Goal Needs

(CORL)

Line Office Needs

32

Sample from Mission Goal Version of Product Matrix

Master, W&W, Climate, Ecosystems, C&T spreadsheets

Products Line Office Mission Goal Review (CORL items in gray)

Motivation: Develop NPOESS products to meet all NOAA user requirements

33

NPP Product Summary

NPP

Totals

Data Records (xDRs/IPs/P3I) 39

NOAA Unique Products 59

Totals 98

34

Acquisition StrategyEmphasis POES Mission Continuity

• Nunn-McCurdy process restructured NPOESS acquisition efforts– POES mission continuity might rely on NPP

data operationally

05 06 07 08 09 11 12 13 14 15 16 17 18 19 20 21 22 23 24 2510

PM N N’ C1 (-)

CALENDARYEAR

NPP

35

Define NPP Products

• POES and EOS Mission Continuity– Essential that we address these products

• Development products– Under development and could go operational

prior to NPP launch

• New products – Never provided to user community and not

under development within NESDIS

36

NOAA-Unique Products Tailored Products

CrIS Thinned Radiances SST Anomalies ATMS Radiances

Total Precipitable Water (ATMS) Coral Reef Degree Heating CrIS Radiances

Snow Cover (ATMS) Coral Reef Bleaching VIIRS Radiances

Precipitation Type/Rate (ATMS) Tropical Rainfall Potential OMPS Radiances

Surface Emissivity (ATMS) Vegetation Fraction Cloud Mask

Cloud Liquid Water (ATMS) Hazard Support (Tropical) Sea Surface Temperature (SST)

Sea Ice Cover/Concentration Hazard Support (Volcano) Ozone Profile

Snow Water Equivalent (ATMS) Total Ozone (CrIS) Ozone Total Column

Ice Water Path (ATMS) Blended Ozone Snow Cover

Land Surface Temperature Outgoing Longwave Radiation Imagery

Temperature Profiles (ATMS) Outgoing Longwave Radiation Ocean Color/Chlorophyll

Moisture Profiles (ATMS) Absorbed Radiation Vegetation Index

Blended Snow Cover Rainfall Prediction Active Fires

Harmful Algal Blooms Tropical Cyclone Intensity Atmospheric Temperature Profile

Regional Ocean Color Tropical Rainfall Potential Atmospheric Moisture Profile

Blended SST Vegetation Fraction Aerosol Optical Thickness

Surface Type & Vegetation Cover

Surface Albedo

Cloud Cover/Layers

NPP Mission Continuity Products

37

NPP Development Products


CrIS Cloud Cleared Radiances Aerosol Particle Size

Clear Sky Radiances (VIIRS) Cloud Top Temperature

Polar Winds (VIIRS) Cloud Top Pressure

Vegetation Health Land Surface Temperature (VIIRS)

Vegetation Moisture

Drought Indices

Vegetation Thermal Conditions

Leaf Area Index

Fire Potential

SST (AVHRR-like)

Aerosol (AVHRR-like)

Carbon products

38

NPP New Products


Integrated xDRs at CrIS Resolution Cloud Base Height

Cloud Liquid Water Path (VIIRS) Cloud Effective Particle Size

Cloud Ice Water Path (VIIRS) Cloud Optical Thickness

Cloud Top Temperature (VIIRS) Cloud Top Height (VIIRS)

Inversion Strength and Height Ice Surface Temperature

Net Heat Flux

Sea Ice Characterization (VIIRS)

Suspended Matter

Atmospheric Pressure Profile

39

Agenda

• Project Overview G. Sloan

• Requirements Mgmt L. Fenichel

• Development Approach E. Wilson

• Tool Selection E. Wilson

8:00 a.m. Welcome





12:00 p.m. Lunch






40

A Reminder

Parts of NPP, NPOESS, and NDE are subject to:– ITARs (International Trafficking in Arms

Restrictions)– “NOFORN” Classification

41

Topics

• Requirements Management

• Development Approach

• Algorithm Development

• Data Handling System

• IT Security

• Data Distribution

• The Way Forward

42

Master Phasing Schedule

43

NDE Baselined DocumentsDocument Status

NDE Concept of Operations Draft Delivered 8/24/2006

Draft Delivered 11/7/2006 (Incorporated RIDS from Peer Review)

NDE Federal Enterprise Architecture

Draft Delivered 8/24/2006

NDE Data Distribution Conceptual Design


NDE Stake Holder Survey Summary Status 2006


Draft Re-Delivered 8/28/2006

NDE Network Services Design Draft Delivered 8/24/2006

NDE System/Subsystem Design Description


NDE Logical and Geographical Connectivity Design


Standards for NDE Algorithm Development, Delivery, Integration & Test


44

NDE Baseline Documents (Cont’d)

Document Status

Project Plan Draft Delivered 8/2004

Draft Delivered 8/2005



NDE Systems Requirement Specification


NDE Configuration Management Plan




Risk Management Plan Delivered 7/14/2006

45

Management Approach

• Staff the NDE Design Team with experienced professionals

• Use design tools• Prototype of Development Environment

– IBM p570 using AIX• Use NPP as the NDE risk reduction mission• Maximize use of COTS and GOTS (minimize

custom code)• Collaborate with similar, current projects• Develop prototypes to validate requirements and

design• Sell-off requirements as early as possible

46

Lessons Learned from NASA’s EOSDIS Core System

• Integrate Algorithms with the Data Handling System early in the life cycle

• Test the system under full operational load – data driven systems behave in strange ways

• Build in Operator Dashboard and System Metrics for reporting– NetIQ and CA’s Unicenter are candidates

• Configuration Parameters – Put in Database for change control, audit trail– Document CPs, min/max values, default, suggested values

• Use “Rolling Wave” technology refresh approach– Constant effort– Keeps budgets flat – no large peaks every 5 years

• Just in Time Hardware buys

47

NDE Design Assumptions

• All NDE users are registered users• Users get data by entering an NDE subscription• Subscriptions have two types

– Standing Order (no expiration date)– Ad-Hoc (covers a limited time period)

• Subscription to NOAA-unique products to be archived to CLASS will be determined by Archive Review Board

• NDE will subscribe to all IDPS xDRs (in lieu of implementing complex logic using IDPS API to order only xDRs needed)

• NDE will not distribute RDRs in near-real time to users (exception: Telemetry)– Users will need to request RDRs from CLASS

• Product latency is customer and product specific

48

Data Access

• NDE will flag customers as authorized for data or not during data denial– Use Initial Joint Polar System (IJPS) Data Denial

Implementation Plan for initial authorization guidance• NDE will maintain a customer database that will

contain sufficient attributes in order to set flag• Restricted data will be delayed for subscriptions

from unauthorized customers– Timeliness more an issue for NPOESS than NPP

• Only authorized users will be notified of data access status– End User License Agreement with customer restricts

redistribution

49

Agenda





8:00 a.m. Welcome





12:00 p.m. Lunch






50

Requirements Management Matrix

ID Section Requirement ContractSRS203 3.14.10 Development Lifecycle Tools - The Contractor will specify

a suite of proven development lifecycle tools to enhance NESDIS capabilities in performing developmental and software maintenance tasks. The documentation will demonstrate that the selected tools are widely supported in the remote sensing software industry and the most likely to be known by future NESDIS support staff. Technologies in this category are: CASE tools, modeling tools, 4th Generation Languages, Testing tools, and Requirements Tracking tools.

SE3

SRS94 3.14.6

SEI Capability Maturity Model - The Contractor shall design the System so that its management capabilities can be evaluated in terms of the Software Engineering Institute's (SEI) Capability Maturity Model (CMM), with the goal of Level 2 certification during proposal evaluation and Level 3 certification three years after contract award.

SM1

SRS201 3.14.8NOAA IT Best Practices - The System shall be designed and built with NOAA IT "Best Practices" guidance from the NESDIS Information Technology Architecture Team (ITAT).

SE8

51

Requirements Management – The Process

• Define Business Objectives• Contained in Contract Requirements Matrices under Desired Outcomes and Performance Standard columns• Additional Objectives detailed in the Concept of Operations

• Automate Requirements Analysis, maintain Traceability (Use DOORS)• Reduce Ambiguity

• Contractual Requirements reviewed, System Requirements Spec (SRS) created

• Derive Requirements to further capture NDE functionality• Use Case Driven Process • Cause/Effect Tables• Automate using UML compliant tools • Domain Expert Reviews and Refinements• Several Iterations

• Capture and Formalize Detailed Requirements • Test Cases generated • ‘Final Round’ of Domain Expert Reviews

52

NDE Requirement Allocations

AllocationsNumber of SRS Requirements

Customer Services 16

Ingest 24

Production 26

Distribution 25

Product Management 22

Common Services (Infrastructure) 50

System Monitoring and Control 22

Security 1

Documentation 12

53

NDE Requirements Traceability

ID Section Short Title Contract AllocationUse

Case

Test

Case

SRS62 3.2.1.3.2 SARSAT Telemetry from IDPS XF1 Ingest UC101 TC25

SRS63 3.2.1.3.3 ADCS Data from IDPS XF1 Ingest UC101 TC26

SRS58 3.2.1.3.4 Product Subscriptions to IDPS

XF1 Ingest UC102 TC44

SRS101 3.2.2.3.1 Available Data Product Projections

PG5 Production UC181 TC124

SRS102 3.2.2.3.2 Available Data Product Aggregations

PG6 Production UC182 TC131

• Maintained in DOORS Tool with Active Links to other NDE Documents• Telelogic Tool Suite (DOORS, TAU Developer)

• Use Cases Mapped• Test Cases Mapped• Test Results Tracked

54

ChallengeRelated

RequirementImpact Mitigation

Maintain Requirements Traceability throughout the Project Life Cycle

(SRS201) 3.14.8 NOAA IT Best Practices

Potential testing gaps and schedule delays as a result

Use of Tools (SRS201/Automated Tool Suite)

Full specification of NDE System Elements and Components

N/A Cost, quality, schedule Employ a methodology (SRS204) where domain experts are involved early in the process, delivery at CDR

Challenges Matrix

55

Agenda





8:00 a.m. Welcome





12:00 p.m. Lunch






56

Development ApproachRequirements

ID Section Requirement Contract

SRS204 3.10.1The Contractor shall identify a widely accepted software engineering methodology to be used on the project.

SE12

SRS94 3.14.6

The Contractor shall design the System so that its management capabilities can be evaluated in terms of the Software Engineering Institute's (SEI) Capability Maturity Model (CMM), with the goal of Level 2 certification during proposal evaluation and Level 3 certification three years after contract award.

SM1

SRS172 3.9.2The Contractor shall evaluate candidate System Elements for operational fitness using Unit, Integration, Regression, Stress (load), and End-to-End System testing.

SE4

SRS173 3.9.2.1

Each algorithm for creating NOAA-Unique products will be described in terms of explicit, expected test results prior to the installation of the NOAA-supplied algorithm on the operational product generation system. The NOAA-Unique algorithms must satisfy these test requirements.

PG8

SRS182 3.11.6

The Contractor shall cooperate with algorithm developers to identify System Test procedures, standards, and the criteria to be applied in determining a system elements' fitness for operational status.

SE4

57

Development Approach Requirements Based Testing

What is it?

Requirements Based Testing (RBT) is a System Implementation Methodology that

Substantially Reduces the Risk of

Building the Right System to the “Wrong” Requirements !

Why are we Concerned about That ?

– Over half all software project defects originate in the requirements phase– 82% of application re-work is related to requirements errors– Only 54% of initial project requirements are actually realized– Requirements problems account for 44% of project cancellations

Statistics by James Martin and others, taken from “Eliminate the Testing Bottleneck”, Borland White Paper, August, 2006, p. 4.

58

The Methodology of Requirements Based Testing

(RBT) It’s about Testing as an Integral Step of Implementation -- not a Separate

Phase

– Design Tests before Designing the Software -- the Test Cases become the Ultimate Statement of the Develops’ Requirements

– Iterate the Requirements-to-Test Case Loop with Users and Domain Experts until they concur that successful Test Execution will produce the result they want – not necessarily what they originally specified!

…to insure that:

• What we Build is What we Say

• and What we Say is What we Mean

59

Traditional Implementation Hand-Off Phases

Natural Language Requirements

Design

Code Test Plans

Natural Language Requirements

Design

Test Plans

• Analysts talk to Domain Experts to express Natural Language Requirements

• S/W Designers Interpret these into Design, which is then interpreted into Code

•Independent Test Team Interprets Requirements to develop Test Plans

The Result: Unresolved ambiguities and inconsistencies between what was developed, what is tested, and what is really needed!

60

RBT Focuses on Formalized Requirements & Test Cases

Formal Req’tsand

Test CaseRepository

Requirements Analysts Users

Domain ExpertsCreate Formal Req’ts from Natural Language

Create Logical Test Cases

Review/Fix Req’ts & Logical Test Cases

Review/Fix/Validate Req’ts & Logical Test Cases

DevelopersTest Experts

Study Logical Test Cases Complete Physical Test Cases

Study Logical Test Cases

Review Design versus Validated Test Cases

Review Code versus Validated Test Cases

62

Sample Technique for Formalized Requirements

REQ’T ID CAUSES EFFECTS

SRS80 System is in Degraded Operations Mode Customer meets Pre-Defined Criteria [TBD] Customer has Pending Data Request

Pending Data is distributed to Customer

Customer’s Pending Data Subscriptions are Ignored while Degraded Operations Mode persists

63

Challenge Related Requirement

Impact Mitigation

Build the NDE Systems as we Say, and Say what we Mean them to Be

SRS94, SRS173, SRS182

Unresolved ambiguities and inconsistencies between what gets developed, what gets tested, and what is really needed!

Utilize Requirements Based Testing (RBT) Methodology

Find a Suite of Development Tools that Support the RBT Implementation Workflow

SRS203 Worst case: develop RBT artifacts manually (using only Excel, Visio, Word) and develop a homegrown database to track baselines, team member updates, and requirements to test traceability

Identify the Development Functionalities needed to Support RBT; establish Criteria to Evaluate Tools’ performance w.r.t the Functions, taking into account interoperability between Tools in a Candidate Suite

Development ApproachChallenges

64

Agenda





8:00 a.m. Welcome





12:00 p.m. Lunch






65

Tool Selection Requirements

ID Section Requirement ContractSRS93 3.8.3.2 The NDE Processing System Design will use COTS and

Open Source software where it is possible, practical, and approved by the Government.

SE10

SRS203 3.14.10 The Contractor will specify a suite of proven development lifecycle tools to enhance NESDIS capabilities in performing developmental and software maintenance tasks. The documentation will demonstrate that the selected tools are widely supported in the remote sensing software industry and the most likely to be known by future NESDIS support staff. Technologies in this category are: CASE tools, modeling tools, 4th Generation Languages, Testing tools, and Requirements Tracking tools.

SE3

SRS215 3.10.2

The contractor shall develop the data processing elements of the future NDE system using the latest proven technologies (programming languages, CASE tools, object repositories, data base management systems, etc.) that are appropriate for remote sensing data processing.

SE13

67

IT Development ToolsFunctional Categories

SUMMARY OF FUNCTIONAL CATEGORY SCORES

1. Interoperability (with respect to a Suite, and with Common Interchange Tools) 0

2. Requirements Specification & Maintenance 0

3. Configuration Management 0

4. Requirements Verification 0

5. Engineering Management/Team Support 0

6. General Architectural Design and Modeling 0

7. Data Modeling 0

8. Business Process Modeling 0

9. Software Analysis, Design, and Construction 0

10. Extensibility 0

11. User Interface 0

12. OS/Network Support - Tool Server 0

13. Tool Administration & Vendor Support 12

14. Documentation, Training, and Ongoing Education 0

15. Vendor Comprehension & Licensing Offer 0

68

Tool Selection Methodology

• Goal: – To Define a Process for Trade-Off Comparisons

between IT Development Tools

• That:– Establishes a Consistent Approach regardless of

the Individuals performing the Evaluation– Produces Quantitative Results for Comparison and

Justification– is Conducted as Objectively as Possible

• . . . and can also be applied to:– Architecture Operational Components– Science Development Tools

69

13. Tool Administration & Vendor Support 12

13.1 Ease of Installation (server/client combined) 50 <none> 0 10

13.2 Degree of Site/User Configurability 50 <none> 0 10

13.3 Vendor includes personal tool setup assistance in the purchase price 90 <none> 0 10

13.4 Number of Release Defects Reported by Vendor 50 <none> 100 0

13.5 Number Un-reported Defects uncovered during Evaluation of Release 80 <none> 10 0

13.6Number of old versions of the tool currently supported 65

Enter the

number: 1,2,3,4… 0 5

13.7 Vendor provides online help support tool 50 <none> 0 10

Sample SectionIT Development Tool Evaluation Worksheet

70

We Attempt to Establish an Evaluation Framework that is

as Objective as Possible• We evaluate each tool with respect to all functional categories, and

develop our own profile of a tool’s capabilities – giving each tool a numeric score for each functionality

• For a suite of tools, we calculate functional category scores by combining scores for all tools in the suite for each category:

Suite Functional Score = Min(Tool Scores) for a “weakest link”

Suite Functional Score = Max(Tool Scores) for a “strongest

component” • Similar approach applied to:

- Science Development Tools - K. Tewari- Architectural Components - A. Al-Jazrawi

Common Function

Specialized Function

71

IBM Rational:Requisite Pro, Clear Case, Clear Quest, Software Architect, (Test Manager)

Telelogic:

DOORS, Synergy CM, Synergy Change, System Architect, Tau Developer, (Rhapsody), … plus Mercury Test Director

Borland:Caliber DefineIT, Caliber RM, Silk Central, Together, StarTeam

IT Development Tool Suites Under Consideration

72

ChallengeRelated


Find a Suite of Development Tools that Interoperate smoothly, allowing us to move from step to step or backwards without manual cut-and-paste transfer of data

SRS203 Vendor Tool Suites are a mix of in-house products and products from buy-outs, not as well integrated as heralded; members of some suites don’t even run on the same set of platforms

Our evaluation methodology makes special provisions to evaluate a Suite as a whole; Tool Interoperability is scored w.r.t the Suite Context; Suite score for that and other common functions is taken to be the “weakest link” of the Suite members

Find a Suite of Development Tools that can be expected to be supported several years into the future.

SRS203 IT Tool development is currently a very dynamic area with many contenders, some of them with outstanding new products

Our evaluation criteria include vendor stability factors, as well as number years experience with the tool (e.g. after a buyout); our down-select list is composed of 3 of the most well-known, stable vendors in this field.

Tool Selection Challenges

73

ChallengeRelated


Business Process modeling tools are closely tied to their vendor’s middleware for collection of information to feed the model; they are not interchangeable to other vendors’ middleware

SRS215 It could be that our first choice for modeling NDE business activities would force us toward middleware that might be inadequate for our near real-time requirements

We will evaluate the middleware architectural components first, as that component will have the broadest impact on overall system performance; however, before making a final decision we will evaluate the Business Process tools to check that we aren’t forced toward an unacceptable product; we may have to balance the gap in tools with the gap in architectural components.

Tool Selection Challenges

74

Agenda

• Context G. Roth

• Environments G. Roth

• Hardware G. Roth

8:00 a.m. Welcome





12:00 p.m. Lunch






75

NPOESS Context Diagram

FNMOC(Fleet NumericalMeteorology and

Oceanography Center)

AFWA(Air Force

Weather Agency)

NAVOCEANO(Naval Oceanographic

Office)

NPOESS(National Polar-orbiting Operational

Environmental Satellite System)(IDPS)

NESDIS(National Environmental

Satellite Data andInformation Service)

(Supports NPP)

(Supports NPP)

76

IDPS Context Diagram

NSIPS(NPOESS Science

Investigator-ledProcessing System)

(cal val)

SDS(Science Data

Segment)(NASA)

CLASS(Comprehensive LargeArray-data Stewardship

System)(archive)

IDPS(Interface Data

Processing Segment)(at NOAA)

ESPC

SAN

NESDIS

NDE

77

NDE Context Diagram

IDPS(at NOAA)

Users

AlgorithmDevelopers

(STAR)(Center for Satellite

Applications and Research)

CLASS(LTA)

(Long Term Archive)

ESPC(Environmental Satellite Processing Center)

NDE

78

NDE Environment Requirements


SRS83 3.14.3The System shall be designed to support Operational, System Test, and Development Environments.

SE2

SRS237 3.14.3.1The Contractor shall provide an Operational Environment design that lowers the cost and risks of generating and distributing NPOESS-derived products to customers.

SE2

SRS85 3.14.3.2

The System shall provide the capability to support the development of the Data Handling System and the development and integration of Scientific Algorithms in a segregated Development Environment.

I3

SRS84 3.14.3.3

During the NPP mission, the System Test Environment will be segregated in a manner that supports "Quasi-Operational" product generation and distribution (Quasi-Operational is defined as 24 X 7 automated product generation and distribution with 8 X 5 Science and Operations support services).

I1

SRS86 3.14.2.3

- During the NPP mission, the System will have the capability to support product volumes of 4 TB/day.- During the NPOESS C1 mission, the System will have the capability to support product volumes of 8 TB/day.- During the NPOESS C2 mission, the System will have the capability to support product volumes of 12 TB/day.

I1

79

NDE System• IT Development Environment

– Where the Data Handling System (DHS) is developed• Science Development & Integration Environment

– Science Algorithm Development Environment– Science Algorithms may be delivered from STAR Collaborative Environment (CE)– Algorithms integrated with Data Handling System (DHS)– Algorithm Performance Tuning– Functional, end-to-end testing of NDE System

• System Test Environment– Stress Testing, Regression Testing and Operations Acceptance Testing– System and Algorithm Performance Testing– Serves as Backup for Operations Environment– Augmented to make products during NPP (quasi-operational)

• Operations Environment– DHS and Algorithms– High Availability– Low NDE Latencies– High Product Volume

80

NDE System

IT DevelopmentEnvironment

Operations Environment(NPOESS)

SystemTest

Environment

ScienceAlgorithm

Development &Integration

Environment




NDEOperational

Environment

NDESystem







NDEOperational

Environment

NDESystem



81

DHS Promotion & Integration Process



SystemTest

Environment

ScienceAlgorithm


Environment


Request forDHS Build

DHS BuildDelivery

2 1

3DHS Delivery

4

DHSDelivery

DHS Promotionto Operations

5

Request for DHSPromotion Delivery

6

82

Algorithm Promotion & Integration Process


ScienceAlgorithm


Environment


SystemTest

Environment


AlgorithmBuild Delivery

2Request for Algorithm

Build

1

3Algorithm Delivery

STARCollaborativeEnvironment

(SCE)

NDE

4

Algorithm Promotionto Operations

Request for AlgorithmPromotion Delivery

5

83

Design

Hardware: Design & Development Environment (Phase 1)

Automated Tool Servers

Dell PE1850 Servers

Dual 3.0GHz

4 GB RAM

2 Win2003 & 1 RHEL4

42U Rack

CITS Administration LAN

Storage

ESPC SAN Expansion

146 GB SATA Drives

2.33 TB (usable)

StorNext FS Server

42U Rack

Processing

IBM p570

16 x 2.2 GHz CPUs

32 GB RAM

AIX 5.3

42U Rack

ESPC Network

StorNext

Development Environment & CM

84

“Rolling Wave” Capacity & Upgrade Planning

IBMp570

N + n

IBMp570

SAN DiskStorage: Ingest &

Distribution

CPUs:Processing& Database

DataDirect

NetworksDisk

Storage

DataDirect

NetworksDisk

Storage

N + n

N + n < 2Nadditional capacityOperations

85

IBM VirtualizationVirtual I/O Server

– Shared Ethernet – Shared SCSI and

Fibre Channel-attached disk subsystems

– Supports AIX 5L V5.3 and Linux* partitions

Micro-Partitioning– Share processors across

multiple partitions– Minimum partition 1/10th

processor– AIX 5L V5.3, Linux*, or i5/OS**

Partition Load Manager– Balances processor and

memory request

Managed via HMC or IVM***

AIX 5LV5.2Linux

Hypervisor

Dynamically resizable

2 CPUs

4CPUs

6 CPUs

Lin

ux

Lin

ux

AIX

5L

V5

.3

Virtual I/O paths

AIX

5L

V 5

.3

AIX

5L

V5

.3

AIX

5L

V5

.3

AIX

5L

V5

.3

Micro-Partitioning

ManagerServer

LPAR 2AIX 5L V5.3

LPAR 1AIX 5L V5.2

LPAR 3Linux

PLM partitions Unmanaged partitions

Hypervisor

PLM agent PLM agent

AIX 5LV5.3

6CPUs

Ethernetsharing

Virtual I/O server

partition

Storagesharing

1 CPU

i5/OSV5R3**

1CPU

* SLES 9 or RHEL AS 4 and above **Available on selected p5-570, p5-590 and p5-595 models***IVM on p5-560Q and below

IVM

86

ChallengeRelated


IDPS requires use of StorNext to write to ESPC SAN vs ESPC security zones

Imposed on NDE by IDPS and ESPC

Multiple NDE hosts also need to “see” the same file system on the ESPC SAN that IDPS uses; therefore NDE is also required to use StorNext as its metadata server

ESPC (Brian Callicott) is pursuing technical solution

Processing multiple read & writes to & from SAN (utilizing StorNext) may not meet product latency requirements

SRS 1333.2.5.4

Data Product Latency Table

CD9

File I/O to/from SAN may not meet latency requirements.

Early testing of StorNext from IDPS to the ESPC SAN and early interface testing of StorNext from the ESPC SAN to NDE.

Potential risk of not having a Product Technical Baseline and model may impact the ability to correctly size the number of CPUs, RAM, Disk, and internal networks required to meet latency requirements.

SRS 1333.2.5.4

Data Product Latency Table

CD9

Insufficient hardware sizing of future System Test and Operations Environments or increased spending to overcompensate for lack of information

Develop a Product Technical Baseline and model with Product Development Lead Tom Schott

NDE Design Challenges

87

Agenda 8:00 a.m. Welcome



10:45 a.m. Building Blocks and Hardware


12:00 p.m. Lunch





• Research to Operations M. McHugh

• SADIE K. Tewari

88

• GOAL: Faster, more efficient transition of research to operations.

• HOW: By facilitating interaction between STAR and NDE developers– Similar:

• Standards; • Configuration management (CM) tools; • CM procedures.

– Common: • Libraries, error handling, development and visualization tools etc.

– Compatible• Architectures.

Research to Operations

89

STAR Collaborative Environment (SCE)

NDE Environments

Software, Libraries and Tools

Standards and Processes

Compatible Architectures

STAR Developers STAR CM

Group

NDE Architect

STAR Configuration Management

Group

STAR IT Advisory

Committee

NDE Design Team

NDE Design Team

Similar Environments

90

Create the Delivered Algorithm Package

(DAP) for an algorithm

Send DAP to the SADIE



Integrated Product Team

91

Create the Delivered Algorithm Package

(DAP) for an algorithm

Send DAP to the SADIE




92

Science Algorithm Development and Integration Environment (SADIE)

Receive DAP from outside of SADIE

Place received DAP under configuration management

(CM) control

Compile algorithm

Baseline the DAP

Ensure DAP has correct format


Integrate algorithm into Data Handling System

Execute algorithm using its DAP

NDE Development

Team

CM Manager

93




(CM) control

Compile algorithm

Baseline the DAP





NDE Development

Team

CM Manager

94




(CM) control

Compile algorithm

Baseline the DAP





NDE Development

Team

CM Manager

95




(CM) control

Compile algorithm

Baseline the DAP





NDE Development

Team

CM Manager

96




(CM) control

Compile algorithm

Baseline the DAP





NDE Development

Team

CM Manager

97

SCIENCE ALGORITHM DEVELOPMENT AND INTEGRATION ENVIRONMENT

(SADIE)



(CM) control

Compile algorithm

Baseline the DAP





NDE Development

Team

CM Manager

98




(CM) control

Compile algorithm

Baseline the DAP





NDE Development

Team

CM Manager

99

Install Algorithm

System Test Environment

Test Algorithm

Promote to Operations

Install Algorithm

Operations Environment

Monitor Algorithm Performance

CM Manager

NDE Development

TeamIntegrated Product Team

ESPC CCB

100

Install Algorithm


Test Algorithm


Install Algorithm



CM Manager

NDE Development


ESPC CCB

101

Install Algorithm


Test Algorithm


Install Algorithm



CM Manager

NDE Development


ESPC CCB

102

Install Algorithm


Test Algorithm


Install Algorithm



CM Manager

NDE Development


ESPC CCB

103

Install Algorithm


Test Algorithm


Install Algorithm



CM Manager

NDE Development


ESPC CCB

104




10:45 a.m. Building Blocks and Hardware


12:00 p.m. Lunch





• Research to Operations M. McHugh

• SADIE K. Tewari

105

Science Algorithm Development & Integration Environment (SADIE)

• Has similar components and configuration as the IT, System Test, and Operational environments

• Uses common CM and Defect Tracking tools as other environments

• Accessible locally and remotely to algorithm developers• Supports development, integration with DHS & test of

new algorithms and those delivered from STAR Collaborative Environment (SCE)

• Supports a suite of software development and science tools (COTS, GOTS, etc.)

• Receives Delivered Algorithm Package (DAP) from SCE

106

SADIE

• SCE users receive

- Access and training in use of SADIE

- Defect Reports• Supports development of additional science tools and

applications (i.e., validation of products, determination of coefficients, etc.)

• Supports trouble-shooting of product generation failures in System Test and Operational Environments

• Supports algorithm functional and regression testing• Supports fine tuning of algorithms for meeting latency

goals in System Test and Operational Environment

107

Candidate Science Development & Integration Tools

• ArcExplorer• ArcInfo• ArcView• CDAT• EDGEIS• Geomatica• GeoTIFF • GrADS• HDF4 Tools• HDF5 Tools

• IDL/Envi• IMSL• MATLAB• McIDAS• netCDF• OPeNDAP• PV-WAVE• SAS• TeraScan• WXP

108

Science Algorithm Development, Delivery, Integration & Test

Standards• Algorithm Development Standards

– Example: Use Similar Development and Test Environments• Algorithm Delivery Standards

– Example: Standardized Algorithm Delivery Package• CM and Defect Tracking Standards

– Example: Algorithm Developers and NDE use same CM and Defect Tracking Tools

• Algorithm Input Standards

– Example: Production Rules Not Hard-coded in Science Algorithms• Algorithm Processing Environment Standards

– Example: Customized Toolkit Containing Utilities• Algorithm Output Standards

– Example: Common File Format for All NOAA-Unique Products

109

ChallengesRelated

RequirementImpact Mitigations

Agreement on Standards…

Adherence to Standards…

SE14 Delays in research to operations

Collaboration w/STAR

• Provide low NDE Latency times for products (minutes)• NDE Latency largely determined by algorithm performance • Algorithms are developed by another part of NESDIS and sometimes in another environment

SRS 10 3.3.3 Fail to achieve major goal of NDE

• Integrate algorithms and DHS as soon as possible• Algorithm Development, Delivery, Integration and Test Standards being developed jointly with STAR• Make STAR Collaborative Environment and SADIE as alike as possible to facilitate Delivered Algorithm Package (DAP)

Design Challenges

110






12:00 p.m. Lunch






111

Agenda

• External Interfaces E. Wilson

• DHS Architecture E. Wilson • Common Services P. MacHarrie

• Subsystem Design A.Al-Jazrawi

8:00 a.m. Welcome





12:00 p.m. Lunch






112

External Interface Requirements


SRS243 3.3.1

IDPS Data Acquisition - The System shall provide the capability to acquire all xDRs, Intermediate Products, SARSAT Telemetry, A-DCS, ancillary, and auxiliary data and metadata from IDPS.

XF1

SRS244 3.3.2External Ancillary Data Acquisition – The System will provide the capability to configure ancillary data acquisition streams from external sources.

PG9

SRS10 3.3.3

Data Product Retrieval - The NOAA-Unique and Tailored Products generated by NDE shall be made available to customers by placement in locations where data can be extracted within a time not to exceed the time specified in the Data Product Latency Table.

XF3

113



SRS137 3.3.4.1

Archive Data Used for Functional Testing - Metadata and ancillary data, as well as Intermediate , NOAA-Unique, and Tailored Products used in documented Functional Tests, will be sent to CLASS.

DA5

SRS138 3.3.4.2Retrieve Data From CLASS - The System will have the capability to retrieve data from CLASS.

DA9

SRS140 3.3.5

MMC Interface Through ESPC - ESPC Operations shall provide an interface between NDE and the NPOESS Mission Management Center (MMC) such that 100% of the NDE inquiries to the MMC and NDE replies to MMC reuests are received by the MMC in a time not to exceed that specified in the ICD, and that 100% of the notifications and inquiries from the MMC to NDE are received by NDE in a time not to exceed that specified by the ICD.

XF9

114



SRS141 3.3.6

Service Requests to IPO - An interface for NDE service requets to the IPO shall be provided such that 100% of the NDE service requests intended for the IPO are delivered to IPO and 100% of the IPO responses to NDE service requests are received by NDE.

XF10

SRS142 3.3.7Service Requests to IDPS - The System shall provide the capability to enable the ESPC/NDE Operations Staff to communicate with the IDPS Operations Staff.

XF11

SRS235 3.3.8ESPC Trouble Tickets - The System shall provide the capability to interface with the ESPC Trouble Ticket system.

SM16

115

NDE Context: Operational Environment (NPOESS)

NDE Work Requests,

Status

NDE

OPS

ESPC

METOP

NPOESS MMC

COOP

Repository

CLASS

IDPS

NDE

Customers

NAVOCEANO

NPOESS Status

MMC Messages

Ancillary Data

Delivery Consolidated

Reports

IDPS Operator Dialogues

xDRs, IPs, SARSAT Tlm, ADCS Data

External Ancillary Data Subscription Requests/

Approvals

xDRs, IPs, NUPs, Tailored Products, ADCS

Data, SARSAT Tlm

Applicant Registration & Customer Support Dialogues

Recovery Datasets Logs

ARWG Approved NUPs

Retrieved Data

PALs

Ordering & Tracking Messages

Customer Profiles

ESPC Operator

Dialogues

Subscription/Status Requests

Status Reports

116

Sample InterfaceOperational IDPS

NDEOPS

IDPS

IDPS Operator Dialogues

Data Orders

Data Delivery Messages

Delivery Consolidated Reports

IDPS Catalog/File Descriptors

ADCS Data

SARSAT Telemetry

Intermediate Products (IPs)

xDRs

117

Summary of External Interface Documents (1/2)

Interface Control Documents (ICDs):• Environmental Satellite Processing Center (ESPC)• NPOESS Mission Management Center (MMC)• Interface Data Processing Segment (IDPS)• External Suppliers of Ancillary Data (METOP, NAVOCEANO,…)• Program Area Lead (PAL) Handbook• Comprehensive Large Array-data Stewardship System (CLASS)• Continuity of Operations Plan (COOP) Repository

118

Summary of External Interface Documents (2/2)

Service Level Agreements (SLAs):• National Weather Service (NWS)• National Ocean Service (NOS)• United States Mission Control Center (USMCC)• United States Global Processing Center (USGPC)• Any Customer Requiring “Denied” Data• Any Customer Requiring System Status Reports

Standard End User License Agreement (EULA)• All Other Customers

Customers’ User Guide• All Customers

119

Challenge Reqt Impact Mitigation

Supporting simultaneous Quasi-Operations as well as System Test modes

SRS84 Customer satisfaction, system performance, operational quality

Staffing plan, design trades

Defining the interface with the CIP

J-I1 Incomplete design Continuance of Operations Plan

Design Challenges

120

Agenda




8:00 a.m. Welcome





12:00 p.m. Lunch






121

DHS Architecture Requirements


SRS93 3.8.3.2The NDE Processing System Design will use COTS and Open Source software where it is possible, practical, and approved by the Government.

SE10

SRS203 3.14.10

The Contractor will specify a suite of proven development lifecycle tools to enhance NESDIS capabilities in performing developmental and software maintenance tasks. The documentation will demonstrate that the selected tools are widely supported in the remote sensing software industry and the most likely to be known by future NESDIS support staff. Technologies in this category are: CASE tools, modeling tools, 4th Generation Languages, Testing tools, and Requirements Tracking tools.

SE3

SRS215 3.10.2

The contractor shall develop the data processing elements of the future NDE system using the latest proven technologies (programming languages, CASE tools, object repositories, data base management systems, etc.) that are appropriate for remote sensing data processing.

SE13

122

Requirements Focus on the NDE Perimeter

page 1

UseCase3

UseCase2

*

*

UseCase4

*

*

*

*

UseCase1

*

*

NDE Perimeter

User Scenarios

Test Cases

123

Defining What’s Inside NDE(Functions, Data, Activities)

ProductFile

defines

is defined by

AlgorithmOutput

creates

is created by

AlgorithmProductInput

read by

reads

Algorithm

defines

instantiates

AlgorithmExecutionInput

readsis read by

AlgorithmExecutionOutput

creates is created byFileMetadata

describes

Product Metadatadescribes

Product

ProductNameQualityStandardProductType

<Undefined><Undefined><Undefined>

File

FileNameQualityDataCreationTimeArchived

<Undefined><Undefined><Undefined><Undefined>

Algorithm

AlgorithmStatusSpatialCoverageArea

AlgorithmExecution

EnqueueTimeCompletionTimeStatus


Metadata

MetadataTypeCoreMetadataAdditionalMetadata


Identifier_1 <pi>

NDE Perimeter

IDEF0

E-R

Activity

124

Overview of the Data Handling Subsystems

1.0 Customer Services

• Customer Subscription Web Service

•PAL Subscription Approval Web Service

• Customer

Status Reporting Web Service

2.0 System Monitoring & Control

• MMC Message Handling

• Process & Component Monitoring & Control

•Mode Switch & Failover Management

• System Logs

• Recovery Datasets

3.0 Product Management

• Algorithm Definition Web Service

• Product Definition Web Service

4.0 Ingest

• Receive & Store Data

• Receive IDPS Delivery Messages & Consolidated Reports

5.0 Production

• Processing Manager

• Production Scheduler (Rule Based)

6.0 Distribution

• User Data Transmission

•Archive to CLASS

• Data Cleanup

7.0 Common Services

• Enterprise Service Bus

•Relational Data Base Management System

•Rule Management

125

Functional Categories for Architecture Operational

Components

• Interoperability• Customer Service Framework• System Monitoring and Control• Product Processing (Ingest, Production, Distribution) Service Framework• Database Management• Business Process Specification• Security Policy Enforcement• Network Availability & Performance Monitoring• Administrative User Interface• OS/Network Support – Tool Server• Component Administration & Vendor Support• Documentation. Training, and Ongoing Education• Vendor Comprehension & Licensing Offer

126

Service Framework

Candidates

BEA:Tuxedo, Weblogic, Aqualogic

IBM:Websphere

Oracle:Fusion

Note: Business Process Modeling capabilities are spread between IT Tools (modeling) & Components (performance data accumulation)

127

ChallengeRelated


Integrating Middleware with CASE tools

SRS215 Maintainability of design Prototyping, early testing, staffing

DHS Architecture Challenges

128

Agenda




8:00 a.m. Welcome





12:00 p.m. Lunch






129

Common Services Requirements ID Section Requirement Contract

SRS89 3.14.1.3

Starting with the NPOESS-C1 mission, the NDE Operational Environment shall not be interrupted for more than 2 hours in any 24 hour period and no more than 4 hours in any 30 day period.

SE2

SRS215 3.10.2

Use Proven Technologies - The contractor shall develop the data processing elements of the future NDE system using the latest proven technologies (programming languages, CASE tools, object repositories, data base management systems, etc.) that are appropriate for remote sensing data processing.

SE2, SE13

SRS159 3.6.2

Scalability - The Operations, System Test, and Development Environments will be scalable--that is, additional capacity (throughput, latency, performance) can be added to it as needed without redesign of the system infrastructure. It will be scalable from the initial system to support NPP up through a system to support the simultaneous operations of NPOESS-C1 and NPOESS-C2.

I1

SRS87 3.2.5.2

Provide Automatic Failover - The System shall provide an automatic failover capability that will re-create a fully functioning configuration from a failed configuration with no more than 1% of the tasks requiring manual intervention.

DD5

130

DHS Architecture

Database

Event Service Bus

Data Handling Subsystems

NDE

Inge

st

Pro

du

ctio

n

DHS GUIs

Dis

trib

uti

on

Product/Customer/System Mgmt

131

Initialization

Database

Event Service Bus

Data Handling SubsystemsIn

gest

Pro

du

ctio

n

DHS GUIs

Dis

trib

uti

on

Initialize:•Algorithms•Product-Generation Rules•Customers•Subscriptions,•System Mode (i.e. Data Access)

NDE


132

Data Driven

Database

Event Service Bus

Data Handling Subsystems

Inge

st

Pro

du

ctio

n

DHS GUIs

Input Events

Output Events

Files Arrive at Landing

Zone

Product Distribution

and Notification

Dis

trib

uti

on

NDE


133

RDBMS Evaluation

• Database (Oracle, DB2)– Scalability, availability, disaster recover and spatial data types

and functions narrows vendors to Oracle, DB2– Highest ranked within industry standard transaction benchmarks

(about 1,600 transactions per second / per CPU)– Both provide clustering approaches for high availability– Both provide disaster recovery using database replication

• Decision– In-house developed spatial benchmark testing the performance

and precision of spatial data types and functions

134

Spatial Benchmark (1 week simulation)

Database

Event Service Bus

Data Handling System (DHS)

Inge

st

Pro

duct

ion

DHS GUIs

Input Events

Output Events

(NPOESS-Like) MODIS,OMI DataMADIS Data

Product Distribution

and Notification

Dis

trib

utio

n

NDE


260,000 xDRs / Day/ Satellite

173,000 NUPs & TPs / Day / Satellite

Ingest * 3Production * 5

433,000 / Day / Sat.

135

Benchmark: File Level Spatial Qualification

• Limit algorithms execution by the spatial area of the input file. (e.g. Coastal and CONUS products)

• Distribute product to customer only when the file’s geographic area intersects the subscription’s geographic area of interest

136

Benchmark: Gazetteer

• “Names” a geographic location• Sources: USGS and National Geospatial-Intelligence

Agency (NGIA) • Purpose

– Serves as an alternate method to specify a geographic filter on a product or subscription

– Provide labels for mapped images (i.e. label ports on products for C&T customers)

137

Benchmark: Event Gazetteer

• Names an Event– Geographic and temporal location of an event

• Event Variable Value or Range– Present Weather ‘Fog’– Present Weather ‘Volcanic Ash’– 24Hr Accum. Precip. 0.175m - .300m

• Sources (EPA, MADIS)• Purpose

– Provides a means of producing or distributing a product in response to an event that has not yet happened, but might in the future.

– Provide additional optional data layers overlapping the geographic a temporal area of the product

138

Benchmark: Pixel Level Spatial Qualification

• Store geo-location data in the RDBMS– Limited Addt’l data (i.e. Cloud Mask, Land/Sea Mask,

Sensor Zenith)

• Purpose– Method of determining whether to produce or

distribute based on Cloud Free in an entire geographic area

– Determining whether to execute blending algorithms– Simplifies the development of new tailored products

(i.e. subsetting) by providing filenames and pixel locations to tailoring tools

139

NDE Data Model

• Product Development View– Supports development of data to define products and “plug”

algorithm into NDE

• Operations View– Supports Ingest, Storage, Archive, Production, Distribution,

System Metrics

• Card Catalog View– Supports ingest, quality control, order specification, archive.

• Science Data View– Supports product tailoring, algorithms, SOA enabler– Alternatives: Relational View, J-METOC, CDM– Selecting: Relational

140

ChallengeRelated


COOP requirements for NPOESS need to be determined

Part of ESPC overall COOP requirements

Needed to identify the details of the database

replication implementation

G. Goodrum resolving this issue

Design Challenges

141

Agenda




8:00 a.m. Welcome





12:00 p.m. Lunch






142

Related Requirements


SRS215 3.10.2

Use Proven Technologies - The contractor shall develop the data processing elements of the future NDE system using the latest proven technologies (programming languages, CASE tools, object repositories, data base management systems, etc.) that are appropriate for remote sensing data processing.

SE2, SE13

SRS169 3.8.3.4Modularity - The NDE System Elements shall be designed so that Scientific Algorithms are invoked as objects.

SE5

SRS168 3.8.3.3Reusability – The NDE System Elements shall be designed to be reused in other Satellite Data Processing applications.

SE2

SRS217 3.10.2.1

Use a 4GL - The Contractor shall develop the data processing elements of the future NDE system using tools that will support the ability to alter executable components without altering source code.

SE14

143

Service Oriented Architecture (SOA) for NDE

• NDE drivers for considering a Service-Oriented Architecture (SOA)– High availability system, need to be able to

update components without bringing system down

– New products and algorithms need to be easily added to the system

– Each launch requires scaling up of the system capabilities

– Future NDE functionality will be easily added as services

144

Service Oriented Architecture (SOA)

• ”Black-box” design - In an SOA environment, resources on a network are made available as independent services with defined interfaces

• In a SOA, processes and services are flexible and can be easily added and rearranged

• The SOA architecture style is geared to provide enterprise business solutions that can change on demand

• The SOA software architecture is based on the key concepts of an application front-end, service, service repository, and service bus.

• Business functionality is provided by services used by application front-ends and other services or even servers

145

SOA Specification

• OASIS (Organization for the Advancement of Structured Information Standards) is a not-for-profit, international consortium that drives the development, convergence, and adoption of e-business standards.

• The OASIS Reference Model for Service Oriented Architecture V 1.0 was approved as an OASIS Standard in 2006.

• Specification available at: http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=soa-rm

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=soa-rm

http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=soa-rm

146

Elements of SOA

Elements of SOA, by Dirk Krafzig, Karl Banke, and Dirk Slama. Enterprise

SOA. Prentice Hall, 2005

147

Developing Services• The following are architectural principles for the design of

services– Service Encapsulation – information hiding and breaking a

program into distinct features – Service Loose coupling - Services maintain a relationship that

minimizes dependencies– Service contract - Services adhere to a communications

agreement – Service abstraction - Services hide logic from the outside world – Service reusability - Logic is divided into services with the

intention of promoting reuse – Service statelessness – Services minimize retaining

information specific to an activity – Scalability – ability to handle growing amounts of work or be

easily enlarged

148

Enterprise Service Bus

Custom/ExistingApplications

Service Orchestration

Presentation and Portals

Data Services

Web Services

Enterprise Service Bus (Routing and Transformation Services)HTTPInternet

DB

149

NDE Data Handling System (DHS) Design Status

• Major System Subsystems/Services have been identified

• Basic use case diagrams have been modeled and used to define much of the problem domain

• Data Flow Diagrams have been created• Entity Relationship Diagrams (ERDs) developed

150

Customer View of NDE

Products are made available

for the user

NDE starts algorithms when requested inputs

for products are available

Products are removed fromthe NDE based on

predefined cleanup rules

Algorithms to produce these products

are defined in NDE

Users select the products they want to receive via a

web interface

Customer works with PAL to define desired products

151

Customer Request View• Data products are defined by a Product Area Lead (PAL)

– Defined tailored and NOAA unique product parameters– Format – Input data sources

• Users log in and subscribe to Data Products (ad hoc or standing)

• User Request View provides access to a list of products available – Optional parameters include

• Spatial coverage area • Data quality thresholds• Compression options• Delivery options – FTP push or pull, etc.• Notification options – web-based GUI planned, other possible

options

152

NDE Subsystems

SystemMonitoring& Control

SystemMonitoring& Control

CustomerServices

CustomerServices

ProductionProduction

ProductManagement

ProductManagement

CommonServices

CommonServices

DistributionDistribution

IngestIngest

153

Customer Services Subsystem

• User Registration Web Service– User accesses registration form via web browser, enters identification

information, and submits request– The user registration request is stored in NDE DB waiting for operator

approval

• User Subscription Web Service– User logs in and accesses subscription web page, selects data for

subscription or ad-hoc request and submits request– Request is automatically approved unless it exceeds configurable NDE

order limits– User can view approved subscription/ad-hoc requests

• User Status Reporting Web Service– The user logs and can receive information on all orders requested by

the user in the system for the past 72 hours– Authorized users can receive system status information

• Customer help requests/ESPC interfaces/roles to be defined in working groups planned in January

154

Customer Services Subsystem Activity Workflow Diagram

155

Customer Services Subsystem Activity Workflow Diagram

156

Customer Services Subsystem

• Registration Approval Web Service– The operator logs into the operator account using a web browser– The operator receives user registration information– Based on input from NDE Manager the operator grants or denies

the registration request

• Subscription Request Approval Web Service– The operator logs into the operator account using a web browser– The operator receives user order information– Based on information from the Product Area Lead (PAL) the

operator grants, edits, or denies the Subscription request

• Archive to CLASS Web Service– Archive algorithms– Archive data – based on guidance from the Archive

Requirements Working Group (ARWG)

157

Customer Services Subsystem Workflow Activity Diagram

158

Customer Services Subsystem Workflow Activity Diagram

159

Product Management Subsystem

• Product Definition Web Service– Web-based GUI allows the different data types handled by NDE

to be defined or modified– XML ingest of parameters currently considered in design– Alternatively, an installation script can be used to enter initial set

of data products• Algorithm Definition Web Service

– Web-based GUI allows processing parameters for algorithms to be entered into the NDE database

– Parameters include• Production Rules to be used in selecting input data• Expected outputs• Expected size of inputs and outputs• Expected processing time

– XML skeleton generation for Algorithm Definition parameters currently considered in design

– XML ingest of Algorithm Definition currently considered in design

160

Product Management Subsystem

Day N

Day N8h

GranuleA GranuleA1

GranuleB GranuleB3

AlgorithmA

GranuleB1

AlgorithmA

GranuleC GranuleCGranuleC

GranuleB2

GranuleC

AlgorithmA AlgorithmA

10h 14h12h

Example Production Rules

161

Ingest Subsystem• Receive and Store Data Service ingests data from IDPS

and other sources– A data delivery message is received from IDPS or other files

received on SAN will have controls written to detect file appearance and initiate the Receive Data Service for processing

– Verifies the data is on the NDE SAN– Performs and enables validation functions – checksum, quality

flag checks as input to problem notifications, file size check– Requests retransmit if file fails checksum/size validations– Inserts granule metadata into NDE database– Inserts spatial metadata into NDE Database

• Receive IDPS Consolidated/Data Delivery Report– Consolidated Data Delivery Report is sent to NDE SAN– ESB configured to generate an event when report is received– Report is compared with granules received by NDE– Missing files are requested from IDPS

162

Ingest Subsystem Activity Diagram

163

Ingest Subsystem Activity Diagram

164

Production Subsystem

• Production Scheduler Service - selects algorithms to execute when data is available– Executes Production Rules– Retrieve needed data from CLASS LTA– Schedule ready algorithms by sending a message to the

Production Server subsystem

• Processing Manager Service – manages NDE algorithms’ execution when their inputs are ready– Runs Tailoring– Reports algorithm execution statistics– Monitors algorithm execution– An algorithm writes data to NDE Production Processing Landing

Zone area to be ingested into the NDE

165

Production Subsystem Activity Diagram

166

Distribution Subsystem

• Archive to CLASS Service sends data to the CLASS • Initiated when a message is received from another Subsystem

– Sends data indicated in message to CLASS– Receives status back from CLASS

• User Data Transmission Service forwards data products to users– Calculates checksum – Compress (gzip, lossy, lossless, etc) data if required – Pushes data to user if required– Stages data to pull location if required– Implements Data Denial

• Data Cleanup Service removes old data from the NDE SAN– Reads NDE DB for products that exceed the maximum cleanup interval

or exceed maximum user defined cleanup interval– Verifies data has been archived to CLASS– Verifies data has been sent to all requesting users– Deletes data from SAN– Updates NDE Database to remove metadata/spatial metadata

167

Distribution Subsystem Activity Diagram

168


169


• Included in Cleanup Rules– Exceeds maximum cleanup interval for NDE– Exceeds maximum user window– NOAA Unique Granules are removed when

• Archived to CLASS• Distributed to Users

– Non-NOAA Unique Granules are removed when

• Distributed to Users

– SAN Full Check will cause removal earlier

170

NDE DHS Challenges

Challenge Impact Mitigation

Validating system actors and their roles

User acceptance Working groups with PALs

Defining web services for use by the PALs, Users, NDE Engineer, and other roles we identify

Ineffective and inefficient web service displays interfacing with DHS

Prototyping

Defining interfaces Performance ICD Working Groups

Transition of legacy algorithms

Cost, schedule, performance

Standards

Defining user notification methods

User needs unmet Working with PALs Customer feedback

171

Agenda

• IT Security B. Callicott

8:00 a.m. Welcome





12:00 p.m. Lunch






172

NDE Security Requirements


SRS160 3.7.1The Contractor shall follow ESPC (DoC/NOAA/NESDIS) procedures and policies for securing ESPC systems.

SA5

173

Security Considerations

• Security is incorporated and addressed from the initial planning and design phase

• Multiple, overlapping protection approaches are used ensuring the failure or circumvention of any individual protection approach will not leave the system unprotected

• NIST 800-27A “Engineering Principles for Information Technology Security (A Baseline for Achieving Security), Revision A” has been followed

• Establishment of a Plan of Actions and Milestones (POA&M) that lays out the resources and milestones to achieve full compliance with NIST 800-53 controls

• Patch Management procedure in development and will be implemented as one of the POA&Ms

• ESPC CCB process has been modified to require ISSO approval of all configuration changes to ensure the system remains secure and documentation is up to date

174

Stage 1 Network Diagram

NSOFEnterprise

DMZ Public

Security Zone

DMZ 2PartnerSecurity

Zone

DMZ 3Development, Pre-Ops, Test

& Integration

Security Zone

Inside Production

IngestSecurity

Zone

Administration

Security &

AdminSecurity

Zone

Security Level 30

Security Level 35

Security Level 55

Security Level 99

Security Level 100

Cisco 6506Cisco 6506Cisco 4507 Cisco 4507Cisco 3745 Juniper FW

VPN ServerCisco 3060 Pair

Cisco 4507

ESPC Secure Network

Temp Fiber

FB-4

File: ESPC Security Zones v15.vsd

Security Level 25

ESPC Firewall

Pair

2nd VendorFirewall Pair

IPS Pair

Switch

IDSSensor

IDSSensor

IDSSensor

IDSSensor

IDSSensor

IDSSensor

Internal-Initiated Traffic to All Zones Except Admin, Following Higher to

Lower Rule

External-Initiated Traffic to Public and NSOF Staging Zones Only

Connections Must Initiate From Higher Zone to Lower Zone (or In-Zone)

System # 5001

Cisco 6506

NSOF

NSOF Staging & Migration

Zone

1Gbps 1Gbps 1Gbps 1Gbps 1Gbps 100Mbps

100Mbps

1Gbps

1Gbps

100Mbps

Fiber-Channel SAN

Outside of ESPC

VPN Security Zone

175

Stage 2 Network Diagram

NSOFEnterprise

DMZ Public

Security Zone

DMZ 2PartnerSecurity

Zone

DMZ 3Development, Pre-Ops, Test

& Integration

Security Zone

Inside Production

IngestSecurity

Zone

Administration

Security &

AdminSecurity

Zone

Security Level 30

Security Level 35

Security Level 55

Security Level 99

Security Level 100

Cisco 6506Cisco 6506Cisco 4507 Cisco 4507Cisco 3745 Juniper FW

VPN ServerCisco 3060 Pair

Cisco 4507

ESPC Secure Network

Temp Fiber

FB-4

File: ESPC Security Zones v15.vsd

Security Level 25

ESPC Firewall

Pair

2nd VendorFirewall Pair

IPS Pair

Switch

IDSSensor

IDSSensor

IDSSensor

IDSSensor

IDSSensor

IDSSensor

Internal-Initiated Traffic to All Zones Except Admin, Following Higher to

Lower Rule

External-Initiated Traffic to Public and NSOF Staging Zones Only

Connections Must Initiate From Higher Zone to Lower Zone (or In-Zone)

System # 5001

Cisco 6506

NSOF

NSOF Staging & Migration

Zone

1Gbps 1Gbps 1Gbps 1Gbps 1Gbps 100Mbps

100Mbps

1Gbps

1Gbps

100Mbps

Fiber-Channel SAN

Outside of ESPC

VPN Security Zone

Cisco 6506VPN Server Cisco 3015 for Dev Only

1Gbps

NDE Development Zone

IDPS

176

VPN Zone

• Associates roles with individuals, allowing individuals to only access certain systems in certain zones

• One subnet of public IPs for VPN on Outside Subnet• Juniper Network Admission Control Appliance (NAC)

for verifying compliance prior to network access, specified for future deployment

• Compliant with US Department of Commerce IT Security Program Policy and Minimum Implementation Standards Revised June 30, 2005

• Compliant with U.S. Department of Commerce Unclassified System Remote Access Security Policy and Minimum Implementation Standards

177

Stage 2 – Development Zone

• ESPC security controls and configuration management processes will apply, one C&A for the ESPC/NDE system

• Removes developers from the National Critical production environment as a risk reduction approach

• Improves collaboration with non-OSDPD developers• Systems dedicated to development – will not be

used operationally • Data acquisition and distribution through drop boxes

in Public Zone• Development programmer access through VPN

Server for Development Only

178

ChallengeRelated


IDPS requires use of StorNext to write to ESPC

SAN vs ESPC security zones

Imposed on NDE by IDPS and ESPC

Multiple NDE hosts also need to “see” the same file system on the ESPC

SAN that IDPS uses; therefore NDE is also

required to use StorNext as its metadata server

ESPC (Brian Callicott) is pursuing technical solution to allow

StorNext FS to run securely in the stage 3 system for file sharing

between zones

Moving from stage 2 to stage 3 requires

implementing some legacy design into the NDE

architecture

NDE architecture will serve as

technical refresh for non-legacy ESPC

systems

Legacy design ESPC systems will require

modification to move into stage 3

POA&Ms will be created and resources assigned to address these legacy

subsystems

Brocade and Cisco Fiber-Channel Switch incompatibilities

ESPC must be able to properly interface

with external organizations over fiber-channel for

data delivery

Unless compatibility issues are resolved,

ESPC may suffer reduced throughput

Initially, Cisco will attempt to resolve incompatibilities;

firmware upgrades must be well-coordinated; long term a single FC-switch

vendor should be selected

NDE Security Challenges

179

Agenda

• Infrastructure Design B. Hickman

• NOAANet Efforts B. Hickman

• Distributed Volumes B. Hickman

8:00 a.m. Welcome





12:00 p.m. Lunch






180

NDE Telecommunications Task Introduction

• NDE telecommunications infrastructure design • Updated status of NOAA migration toward a One-NOAA

NOAANet• NDE Distribution volumes to users 2007-2020 and Section

Summary

ID Section Short Title Contract

SRS216 3.2.3.4.1 Trade Study Alternatives CD5

181

Risk Reduction Telecommunications Options

• Flexible Extensible Switch Based NDE Design Options scale multiple selectable layers of redundancy for COOP – Each of the 4 Options offer progressively increasing:

• component redundancy (power supply, board, chassis, switch, and WAN) to seamlessly meet operational failover contingencies

• scaled firewall protection and security context organization, aggregation, and management

• Maintains optimum compatibility with ESPC Operations:– ESPC Consolidation of IT Architecture– IPV6 Migrations

186

Agenda

• Infrastructure Design B. Hickman


• Distributed User Volumes B. Hickman

8:00 a.m. Welcome





12:00 p.m. Lunch






187

Data Distribution Design Challenges

• Stakeholder Interviews, Model creation, & PAL product profiling

• Current ESPC not able to handle NDE order of magnitude increased loads

• DoC and NOAA security requirements limit design options, particularly the SAN

• Consolidated approach needed to develop infrastructure to handle NDE distribution to users

188

Data Distribution - WAN

• NOAA has 12 legacy hub-and-spoke networks• Accelerate migration to a unified NOAANet to

achieve increased capacity, MPLS any-to-any topology, cost-efficiency

• Washington DC MAN recently upgraded by NWS Qwest solution – DWDM high speed ring with access to distributed

sites via MPLS Cloud

• Leverage Internet2 possibilities• NWS, NEXRAD, & CLASS provide precedents

for NDE design consideration

189

Overview of August 2006 Peer Review Slides

35 Connectors (connect directly to the Abilene network) 26 International Peers 6 Federal Peers (DREN, Esnet, NISN, NREN, vBNS, USGS) 244 Participants 147 Sponsored Participants

Source: http://abilene.internet2.edu/maps-lists/

Abilene 10 Gbps NetworkLambdaRail Network (LRN)

• Slides depicting Internet2 opportunities – Lambda Rail MBone could support Multicast and other Real time

protocol studies for NDE

190

Legacy Washington DC MAN

Wayne AvenueSilver Spring

FOB Suitland

NSOFSuitland

WWBCamp Springs

Colesville RoadSilver Spring

Landover

NASA GSFCGreenbelt

NOAA HQSSMC

Gaithersburg

HCHB

Germantown

U N I V E R S I T YU N I V E R S I T Y

Univ. of MDCollege Park

(NOAA MAXGate)

TLS Domain 26Verizon

TLSDomain

287Verizon

All 10 Mbps

OC12Qwest

Internet Abilene/Internet2

Based on information in Statement of Need DG1330-05-RP-1038, Amendment 0004, Section C

10 Mbps

100 Mbps

1000 Mbps

1000 Mbps fiber (leased from Fibergate)

Circuits likely to need capacity upgrade for

NDE volumes

191NWS Logical Network Design in 2006

NWS 2006 Upgrade to Washington DC MAN

NSOF

192

NWS 2009 Projected Growth

NWS Logical Network Design in 2009

NSOF

193

Agenda

Infrastructure Design B. Hickman


• Distributed Volumes B. Hickman

8:00 a.m. Welcome





12:00 p.m. Lunch






194

August 2006 Peer Review Update

• Large number of slides showing Legacy and Nextgen satellite comparison charts

• Projected volumes did not include: Latency Overhead Surge Firewall delay Encryption delay Compression delay

• Distributed Delivery slides to follow do reflect (all but firewall delay) scaling for throughput

• Nunn-McCurdy changes revising estimate basis and model refinements

195

OverArching NDE Models Basis

• xDR (RDR, SDR, EDR, & IP) sizing model establishes IDPS delivery to NOAA NDE SAN

• xDR model outputs drive NOAA Product model inputs

• NOAA Product model tracks legacy/next generation traffic growth and required distribution to identified End-users

• Orders of magnitude larger than legacy systems

• Large number of significant distributed users

196

Predicted Domestic Traffic

NDE Customer/Stakeholder

Customer Destination Location

(All distribution originates from NSOF)

2007- 2009

Traffic (Mbps)

2013 Traffic (Mbps)

2015 Traffic (Mbps)

2018 Traffic (Mbps)

2020 Traffic (Mbps)

NOAA/NOC Silver Spring, MD 625 717 1,373 2,875 4,083 NWS/TOC Silver Spring, MD 466 534 1,023 2,142 3,042NWS/NCEP Silver Spring, MD 563 645 1,236 2,588 3,675NWS/AWIPS Silver Spring, MD 369 423 810 1,696 2,409ORA (NCWCG) College Park, MD 37 43 81 171 242ESPC (Development@NCWCG)

College Park, MD 375 430 824 1,725 2,450ESPC (USMCC <SARSAT & A-DCS>) NSOF Suitland, MD 72 137 288 408NDE CIP TBD 287 549 1,150 1,633CLASS NSOF Suitland, MD 365 418 801 1,677 2,382

NMFS Silver Spring, MD 25 29 55 115 163

NOS Silver Spring, MD 375 430 824 1,725 2,450NOAR Silver Spring, MD 135 155 298 623 885NMAO Silver Spring, MD 68 78 149 311 442Dept. of Agriculture Washington, DC 25 29 55 115 163FAA Washington, DC 68 78 149 311 442Dept. of State Washington, DC 25 29 55 115 163TOTALS NSOF Suitland, MD 3,520 4,394 8,419 17,627 25,032

200

Trade Outcomes

• Commercial satellites are band-limited • MPLS savings realized by distributed, low bandwidth

networks (T-1 and below OC3) in range of 25%-40%– SONET still found to be the most cost effective for distributed

networks of OC3 rates and above Multicast or alternative ‘One-Many’ technologies must be

established at customer edges and in the commercial clouds to reduce NDE redundant loads

Dense Wave Division Multiplexing (DWDM) can run very large (~40 Gbps) traffic volumes simultaneously over the same (1Gbps) fiber land lines

201

Bandwidth Reduction Recommendations

IP multicast• Lossless compression• Lossy compression• IBM General Parallel File

System (GPFS) Real-time protocols Threaded & other highly

parallel processing methods

• Reduced resolution requests

• Reduced aerial coverage requests

• Reduced frequency of requests

• Updating (transmitting) only the ‘new data’ for composites

• Increasing product requests over data requests

Technologies Strategies

202

Summary• NDE Telecommunications Design for NSOF

flexible and extensible to scale appropriately and meet all throughput requirements

• Current NAC feedback:– NDE & NWS can add capacity to new Qwest DWDM ring – NDE to apply for HPCC prototyping of Internet2 (Abilene &

Lambda Rail)• Real time protocols, one-to-many (Multicast) solutions, and enabling

technologies prior to C1 launch• Work with NCEP (model assimilation), vendors (e.g., CISCO, IBM,

Oracle, etc.), & COTS providers (e.g., ESRI, ITTVIS, SSEC, Mathworks, etc.)

– Specific NAC solution recommendations to NDE regarding connectivity to distributed National & International end-users expected by March 2, 2007 (NLT June 2007)

203

ChallengeRelated


To define NOAA Enterprise solution

3.2.3.4.1 - Trade Study Alternatives

Higher cost, lower performance, schedule risk

Outreach - OSD, ITAT, & NAC

NAC to recommend solution by 3/2/07

High NDE Data/Product Volumes & Large Distributed customer base

3.2.3.4.1 - Trade Study Alternatives

Unable to meet customer product requirements

HPCC prototype study

IP multicast

Real-time protocols Threaded & other highly parallel processing methods

Location of CIP 3.2.3.4.1 - Trade Study Alternatives

Single point of failure AFWA, CDA-W alternatives under consideration – TBD before C1 Launch

IPV6 Migration 3.2.3.4.1 - Trade Study Alternatives

Inadequate performance HPCC protocol & one-to-many prototype studies

NDE Telecommunications Design Challenge Mitigation

204

Agenda

• What’s Next G. Sloan

• Technical Challenges G. Goodrum

• Closing Remarks J. Silva

8:00 a.m. Welcome





12:00 p.m. Lunch






205

The Road to CDR• We have a resource-loaded schedule and an EAC• Maintain current staffing through CDR• NPP Build Cycle: Development Environment (PDR)

– IT Development Environment– Science Algorithm Development & Integration Environment– Configuration Management Control Processes– Tools Analysis and Selection– Product Technical Baseline

• NPP Build Cycle: System Test Environment (post PDR)• Document updates and baselining

– CM Plan– Concept of Operations– QA Plan– Project Plan– System Requirements Specification– System/Subsystem Design Specification for NPP– FEA Mapping

• Prototyping– Subscription– Ingest– Production

206

Agenda




8:00 a.m. Welcome





12:00 p.m. Lunch






207

Technical Management

Challenges Strategies

NetworksVolume, widely distributed customers, low latency, diversity and inadequacy of current networks

Develop ‘enterprise’ solutions in collaboration with AWIPS and NAC

Testing

- Different communities (IT, Users, Science Developers)

- Design (algorithms as objects supported by IT elements)

- ‘Requirements Based Testing’ for IT

- PALs to work with science users to develop acceptance criteria and test specs

System DesignTendency to prototype and implement too early in life cycle

Quality Management: Monthly government review of design work products

Security PolicyChanging policy over the project life cycle

Work with ESPC ISSO, IT Security Team, and ESPC management

External Interface

- SAN switch interoperability

- Customer Readiness

- Work with NPOESS External Interface Working Group

- Establish “Systems Integration Manager” to coordinate transition teams by product line

208

Agenda




8:00 a.m. Welcome





12:00 p.m. Lunch






209

Program ManagementChallenges Strategies

Transition to Operations

- Continuity of current operations- Exploitation of new products

Establish “Systems Integration Manager” to coordinate transition teams by product line

Stakeholder

Comm.

NESDIS, Line Offices, Other Centrals, Standards Bodies, Goal Teams, Budget Examiners

Monthly Team Meetings with NESDIS, NPOESS & Centrals Working Groups, Product Workshops for Goal Teams & Line Offices, actively seek CIO & ITAT guidance, dedicate resources to budget exercises

Responsibility Transfer

- IT developers to IT operators and maintainers

- Algorithm developers to algorithm maintainers

Develop new estimating and scheduling

procedures with STAR & OSDPD

Budget - FY08 deficit, FY09 threatened

- Flat-lined thereafter

- Prioritize FY08/FY09 allocations & redirect resources accordingly

- Continue to seek plus-ups for product development

ScheduleNPP/NPOESS program milestones are NDE constraints

Plan for readiness ~ 6 months prior to NPP/NPOESS milestones

Resources

- HR: Contractors dedicated to NDE - Infrastructure: Potential underutilization (e.g. NPP/NPOESS delays)

- HR fold contractors into operational support

- Infrastructure (scalable, extensible architecture)

210

Review Calendar

2006 Date Deliverable TaskedNov 21 Requests for

Action (RFA)Review Board

Dec 6 Action Responses to

Board Chairperson

NDE

Design Team

Dec 15 Final Report Design Board Chairperson

211

Happy Thanksgiving!

212

Agenda

• RFA Review V. Griffin

8:00 a.m. Welcome





12:00 p.m. Lunch






npoess data exploitation (nde) preliminary design review november 21, 2006 nsof building 4231...

Documents

rfas rfas

pdr board review of

rfas board members

room rfas

silva slide

review board member

schott slide

slide number