ocean observatories initiative eoi user review june 16 2011 ooi cyberinfrastructure release 1 scope...

Ocean Observatories Initiative

EOI User ReviewJune 16 2011

OOI CyberinfrastructureRelease 1 Scope

OOI CI - EOI User Review16 June 2011


OOI Integrated ObservatoryObservatory Requirements:

Provide one integrated observatory interface to all users inside and outside the

OOI. Enable the users to investigate observations,

manage the observatory and its assets and collaborate with

each other in teams.Engineering Drivers:

A geographically distributed system of systems with

observatories at multiple scales and operational

authority. OOI-wide need for data distribution, storage,

processing and command and control.


Observatory Requirements:Enable communities of users to work with the instruments and observatory assets they need from across all OOI sites in a

uniform observatory environment. Support multiple

“virtual” observatories in parallel.

Engineering Drivers: Multiple heterogeneous

communities of users accessing shared observatory assets

across the OOI. Enable access to observatory assets based on

user authorization level and observatory policy.

Virtual Observatories

Obse

rvato

ry

Inte

rface


OOI Deployment TopologyObservatory RequirementsProvide observatory assets to

support measurements of scientific processes. Provide computational and network assets to enable real-time,

secure access and processing for the observatory users

Engineering Drivers: Proximity to observatory assets, geographic redundancy. High-bandwidth, low latency access to national and international

network peering points. Access to commercial and academic

compute clouds


National & International Observatory Integration

NEPTUNE Canada

NSF OOI Network

NOAA N-Wave Network

Global LambdaIntegrated Facility

Engineering Driver:The topology used by the

OOI program is easily integrated into national and international cyber-

networks. This will facilitate collaboration for

developing the international ocean observing network


Network Messaging Infrastructure

•Based on the Layer 2 National Network Infrastructure•Provides direct application-level messaging connectivity

between components (Capability Containers) within the Integrated Observatory Network, and at users’ sites.


What's Different about OOI?

Integrated Observatory Network Integrated Observatory Network


Event Detection and Response

Observatory Requirements:Support closed loop scientific activities.

Enable interactive and event-driven control of the observatory assets based

on interactive and real-time observations and analyses.

Engineering Drivers: Workflows accessing real-

time measurements, external data sources, real-

time processing, QC and event detection, complex numerical models leading to control of mobile and stationary observatory

assets.


Integrated Observatory Network

Integrated Observatory Network

Integrated Observatory Network: 4 Releases

R2

R1

R3R4

R4: Interactive Ocean Observatory

R2: Managed Instrument Network

R1: Data Distribution NetworkR3: On-Demand Measurement

Processing


Release 1 of ION: Data Distribution Network

•publish data to ION• register data sources

•find and browse data resources•examine data resources

• initiate data notifications and subscriptions•download data from system•get live data streams

•create user accounts•scale computing resources•perform diagnostics and monitoring• receive events without delay

'User-visible' Highlights



•“Ingest” data products from external data sources and experimentally from sensors in canonical and raw formats

•Characterize data sources with their metadata attributes (format, structure)

•Distribute data via streaming and DAP servers to consumers such as data analysts, numerical modelers

•Provide a platform for instrument integration development, with instrument control and sensor data acquisition

•Provide a distributed service integration and execution platform in more than one programming language

Technical Perspective



•Data distribution via publish/subscribe•messages received are immediately distributed•processes act on messages as soon as received (workflow!)

•Scalability: manage growing amounts of work in a graceful way•new process host (virtual machines) can be started as needed•sets up for automated compute resource management

•Fault tolerance: continue operation when part of system fails•can restart a processing unit, pick up where previous left off

•Security: protect information from misuse, theft, or corruption•basic user and policy mechanisms to enable core model

Structural Highlights


Release 1 Components

https://confluence.oceanobservatories.org/display/syseng/

CIAD+AV+Transition+to+Operations


CI Components Developed

https://confluence.oceanobservatories.org/display/syseng/

CIAD+AV+Transition+to+Operations

• Python Capability Container• Java Capability Container Access Library• Web UI Platform• Exchange Messaging System• Distributed State Infrastructure• Resource Registry Framework• Data Publish-Subscribe Framework• Event Notification Framework• Science Data Persistence and Transport Format• Instrument Agent Framework• Elastic Processing Unit• Virtualized Cloud Management Tools


Use Cases for Release 1

UC.R1.05 Synchronize State Data

UC.R1.06 Distribute Data Product

UC.R1.07 Subscribe To Data

UC.R1.08 Persist Streamed Data

UC.R1.01 Hello User

UC.R1.02 Hello Instrument

UC.R1.03 Hello Data Source

UC.R1.18 Command An Instrument

UC.R1.20 Command A Resource

UC.R1.04 Ingest and Describe Data

UC.R1.21Derive Data Product Externally (Merge Data)

UC.R1.09 Discover Resource

UC.R1.14 Use Service Anywhere

UC.R1.15 Put Services Anywhere

UC.R1.16 Scale the Processing

UC.R1.17 Replicate Service

UC.R1.25 Assure Reliability

UC.R1.28 Operate System

UC.R1.29 Monitor System

UC.R1.30 Troubleshoot System

Data

Services &Operations

Basic Connections


"Hello User" Use Case1.User registers with the system by providing an (external) identity, creating an

internal identity for user.

• Internally, user has an internal identity and a user "profile"; which contains 'life cycle state'

of the user and other user attributes such as contact information and preferences.

• User authenticates to external identity provider, in order to assert external identity

• Background: The InCommon federation facilitates the exchange of knowledge that enables

the assertion of identities across organizations within this federation. The NSF CI-Logon

project is developing cyberinfrastructure software to apply InCommon processes within CI

systems, such as OOI. A user, for instance from UCSD, will be able to use her organization

credentials to register with OOI, and subsequently use OOI credentials.

• Each user identity is tracked in the user identity repository.

2.User logs in to the system using identity credential (e.g., login ID and password).

3.User accesses the system's web user interface, which calls system services.

• Features: Status, Register Resources, Search (User, Instrument, Data, Process), Instrument

Access

4.User is assigned their default role in system


Implications of OOI Approach

• We have user identities in the system, enabling security

• We keep user attributes of interest to us (e.g., contact info)

• We assign attributes as needed (e.g., user roles)

• ION doesn't perform authentication itself, so …

• Users don't have to remember a custom OOI identity

• ION doesn't have to maintain user credentials internally

• passwords and associated security issues

• support for all the associated human issues

• software we don't have to write, or maintain

• This solves a big scaling issue


Select Identity Provider and Log On

Cornell UniversityDuke UniversityGeorgetown UniversityGoogle


Identity Provider Logs You In

We get a certificate saying who (Google says) you are.


Data Distribution Use Case1. Data is accepted at a specific Acquisition Point CyberPoP or any specific capability

container.

• External data will be accepted via an Instrument Agent or other adapter process, and go through an

acquisition process before ingested as complete data set or increment for a data set.

• Information does not have to be ingested (i.e. made available in the resource registry, normalized)

to be distributed via the Data Distribution Network

2. Once minimally described, data is sent to the Exchange by the Data Producer.

• Ideally data should be fully described before routing, not just minimally described.

• Further description of the data may already be present in the resource registry of data sets.

3. The Exchange routes data messages to all subscribers to a data stream or filter that match

certain criteria in the description of the data message

1. Matching certain criteria is also fundamental to several other use cases, including explicitly in

Version a Resource.

• Subscribers equally see data, whether internal (local or remote) or external users.

• A data store (or several different stores) is a subscriber to most forms of ingested science data, in no

way different to any other subscriber. In case there is no data store subscriber, data will be

distributed across the Data Distribution Network without any form of persistence; subscribers are

solely responsible for maintaining the information they need.


Subscribe to Data Use Case1.Data Consumer identifies material of interest.2.Variant A: Data Consumer selects existing data stream that provides

the material of interest.3.Variant B: Data Consumer selects historic data set resource …4.Data Consumer registers (through API or web form) with Subscription

service to be delivered material of interest, per a defined set of criteria.5.Data Consumer specifies subscription delivery modality with

Subscription service.6.Registration is acknowledged via message returned to Data Consumer.7.Registration is verified through inspection of the Inventory and Data

Stream Registry.8.Data messages meeting registered criteria arrive within Exchange,

where they are routed to subscribed Data Consumer.9.Event notification messages generated from event producing

processes (e.g. Instrument Agents, Data Source adapter, event aggregator), arrive within the Exchange, where they are routed to registered Data Consumer.


Implications of "Data Distribution"

• User gets data when we get data (no waiting!)

• Users or modules can subscribe to selected data

• Processing routines kick off when data arrives … workflows!

• Fundamental artifacts are data streams (not static sets)

• ION internalizes the data in its constituent elements

• Allows fully using the data's natural structure

• Can present the data according to users' needs

• All kinds of data sources: models, sensors, observatories,

people, …

• ION can keep a 'reference copy' for fast, reliable access

• The data, and products, can be well curated (good metadata)


CI System Design Engineering Milestones

• Passed Life Cycle Objectives (LCO) Review for Release-1

(Mar 2010)

• Approval of Release-1 system architecture baseline and use cases

• Demonstration of critical technologies for risk mitigation

• Passed Life Cycle Architecture (LCA) Review for Release-1 (Aug 2010)

• Approval of refined Release-1 use cases and system architecture for

Construction

• Demonstration of an end-to-end running system and a scalable architecture

• Passed 2 subsystem LCA reviews: Instrument Platform Agent Architecture,

External Observatory Integration (December 2010)

• Passed Initial Operational Capability (IOC) Review (May 2011)

• Up-to-date approved architecture and design

• Added CyberPoP and network design; user experience and user interface

designs


CI Development Progress by Subsystem

ID Subsystem# R1 L4 Req’s

Satisfied

# R1 Use Cases

Satisfied

% Design complete

SLOC# Tasks

complete

SA Sensing & Acquisition 3 (7%) 3 (75%) 39% 1,198 30

AS Analysis & Synthesis N/A N/A N/A 0 0

PP Planning & Prosecution N/A N/A N/A 0 0

DM Data Management 41 (59%) 5 (56%) 95% 23,139 110

COI Common Operating Infrastructure 31 (26%) 4 (36%) 95% 34,467 155

CEI Common Execution Infrastructure 14 (56%) 8 (100%) 77% 35,599 109

IPA Instrument and Platform Agents 14 (41%) 1 (100%) 100% 11,587 52

EOI IOOS Integration Package 1 15 (100%) 1 (100%) 100% 7,793 24

UX User Experience N/A 0 (0%) 80% 24,701 67

ITV Integration, Test and Verification N/A N/A 100% 6,984 N/A

TCN Terrestrial CyberPoPs and Network N/A N/A 100% N/A N/A

41


EOI Presentation

ocean observatories initiative eoi user review june 16 2011 ooi cyberinfrastructure release 1 scope...

Documents