MBARI’sMBARI’sShore Side Data SystemShore Side Data System

FromFromShips, ROVs, Moorings, AUVs, & ?Ships, ROVs, Moorings, AUVs, & ?

ToToBytes, Plots, Pictures, Samples, & Bytes, Plots, Pictures, Samples, &

VideoVideo

21 Nov 2002

2

What Are Our Goals?What Are Our Goals?

Build data systems that can grow over time Build data systems that can grow over time Make adding instruments and data routineMake adding instruments and data routine Easily add new and unimagined components Easily add new and unimagined components Scale to meet growth needs of observing systemsScale to meet growth needs of observing systems

Create an extensible IT umbrellaCreate an extensible IT umbrella Encompass the real world of data sourcesEncompass the real world of data sources

shipboard and shore-side systemsshipboard and shore-side systems isolated buoys and networked observatoriesisolated buoys and networked observatories one-off data files and high-speed (Gb net) streaming dataone-off data files and high-speed (Gb net) streaming data

Embrace image, video and document archive Embrace image, video and document archive formatsformats

Still provide users with “do what I want”Still provide users with “do what I want”

21 Nov 2002

3Cruise (Expedition) Cruise (Expedition) InterfaceInterface

21 Nov 2002

4

Samples InterfaceSamples Interface

21 Nov 2002

5

Video Annotations InterfaceVideo Annotations Interface

21 Nov 2002

6

3D Visualization Interface3D Visualization Interface

21 Nov 2002

7

What Has MBARI Learned?What Has MBARI Learned?

Metadata: It Metadata: It mustmust accompany the data accompany the data Data w/out metadata is like a directory with no ReadmeData w/out metadata is like a directory with no Readme A system’s power relies on good knowledge of its dataA system’s power relies on good knowledge of its data

Metadata: It Metadata: It mustmust accompany the instrument accompany the instrument Every connector between the two increases error ratesEvery connector between the two increases error rates Once data and metadata detached, reattaching is painfulOnce data and metadata detached, reattaching is painful

Metadata: It must be flexible and yet structuredMetadata: It must be flexible and yet structured Flexible: you’ll need to define new kinds of data sourcesFlexible: you’ll need to define new kinds of data sources Structured: consistency => automation => valueStructured: consistency => automation => value

Distributed storage, great interfaces, transparencyDistributed storage, great interfaces, transparency

21 Nov 2002

8

About MOOS: MBARI’s About MOOS: MBARI’s Ocean Observing SystemOcean Observing System

A major observing platform development A major observing platform development initiativeinitiative Multi-platform, cabled & uncabled, benthic to surfaceMulti-platform, cabled & uncabled, benthic to surface

““What Would It Take?” — develop and test ideasWhat Would It Take?” — develop and test ideas Need a way to store metadata with instrumentNeed a way to store metadata with instrument Need a way to submit metadata to data systemNeed a way to submit metadata to data system

Result: Answers that can work anywhereResult: Answers that can work anywhere Local ‘intelligent’ storage: PUCK conceptLocal ‘intelligent’ storage: PUCK concept Consistent services: Instrument SW InfrastructureConsistent services: Instrument SW Infrastructure

We can iterate to good, tested solutionsWe can iterate to good, tested solutions

21 Nov 2002

9

MOOS MOOS (Showing Data Flow)(Showing Data Flow)

Devices

DeployedPlatform

Shore Side Data System

User

Applications

(User Tools)

Data Tracking

Communications

Applications/ Interfaces

Archiving

101110110011

110234999

DataPresentation

Data line 1more datalast data

OceanOceanSideSide

ShoreShoreSideSide

            Portal

21 Nov 2002

10

About SSDS: The About SSDS: The Shore Side Data SystemShore Side Data System

A MOOS Development ProjectA MOOS Development Project Goals: low cost, flexible, expandable, reliableGoals: low cost, flexible, expandable, reliable Future systems beyond MOOS (e.g., MARS)Future systems beyond MOOS (e.g., MARS) Now in 3rd year, deploying initial elementsNow in 3rd year, deploying initial elements

Key Tenets of SSDS DevelopmentKey Tenets of SSDS Development Iterative development—improve it as we goIterative development—improve it as we go Test with real data—new and archivalTest with real data—new and archival Build for change—use modular interfacesBuild for change—use modular interfaces

21 Nov 2002

11

Shore Side Data SystemShore Side Data SystemRequirementsRequirements

Ingest data in Ingest data in any described formatany described format and save it and save it Capture, publish data descriptions Capture, publish data descriptions (metadata)(metadata) Provide Provide standards-based accessstandards-based access to data to data

Raw data, and other common digital formatsRaw data, and other common digital formats APIs for common visualization and analysis toolsAPIs for common visualization and analysis tools User-oriented web interfaces, quick-look plotsUser-oriented web interfaces, quick-look plots

Merge data (different sources & time intervals)Merge data (different sources & time intervals) Support data visualization & quality controlSupport data visualization & quality control Provide data access security as neededProvide data access security as needed

21 Nov 2002

12

SSDS ElementsSSDS Elements

Applications

Data Presentation

Data line 1more datalast data

Ingest

110234999223207191

Archiving

101110

110011

Arriving DataData Tracker

Data Catalog External

Data Stores

Data For Analysis

Web I/F< Requests

Data >

SharedDescriptions

< R

eque

sts

Dat

a >

Metadata

(Re)Processed and New Data Sets

Automated Data FlowInternal InterfacesOn-Demand Interactions

21 Nov 2002

13

How Does It Work?How Does It Work?1.1. First, the developer describes what’s in an First, the developer describes what’s in an

instrument’s data records (instrument’s data records (the metadatathe metadata).).2.2. That description is stored with (or near) an That description is stored with (or near) an

instrument, and sent to SSDS before any data instrument, and sent to SSDS before any data from the instrument. SSDS tracks this info….from the instrument. SSDS tracks this info….

3.3. Data records generated include the Data records generated include the data typedata type..4.4. SSDS automatically routes data of each type to SSDS automatically routes data of each type to

the correct ‘data bucket’.the correct ‘data bucket’.5.5. SSDS automagically knows about the data, SSDS automagically knows about the data,

because they’ve been described. Now it can:because they’ve been described. Now it can: PlotPlot Print Print Search Search Merge Merge Format (on request)Format (on request) Describe (in files & Describe (in files &

headers)headers) Send to ApplicationsSend to Applications Point to Point to by variable by variable

namename

1 2 3 4

21 Nov 2002

14

Standard Standard InterfacesInterfaces

Applications

Data Presentation

Data line 1more datalast data

Ingest

110234999223207191

Archiving

101110

110011

Arriving DataData Tracker

Data Catalog External

Data Stores

Data For Analysis

Web I/F< Requests

Data >

SharedDescriptions

< R

eque

sts

Dat

a >

Metadata

(Re)Processed and New Data Sets

DODSLAS

XML/DTDs

HTTP

Z39.50

SQL

netCDF

FGDC

inGridBrowsers

andand Tools Tools

ncBrowse

DODS

Ferret

21 Nov 2002

15

Strategies (Hard-Earned)Strategies (Hard-Earned)

Low threshold for user entry (minimal XML)Low threshold for user entry (minimal XML) Stay away from domain-specific solutionsStay away from domain-specific solutions

Example: ‘deployment’ is useful conceptExample: ‘deployment’ is useful concept Minimize internal structure & assumptionsMinimize internal structure & assumptions

Our biggest challenge: flexible architectureOur biggest challenge: flexible architecture Be agnostic about input data & file formatsBe agnostic about input data & file formats

Maximize access/presentation featuresMaximize access/presentation features Provide many views into data (common first)Provide many views into data (common first)

Take advantage of OO methods and reuseTake advantage of OO methods and reuse

21 Nov 2002

16

SSDS Multi-View Interface SSDS Multi-View Interface

21 Nov 2002

17

SSDS Multi-View Interface SSDS Multi-View Interface

21 Nov 2002

18

Data Integration StrategyData Integration Strategy

Data can be remote (managed by links)Data can be remote (managed by links) Domain-specific tasks done externallyDomain-specific tasks done externally

Domain-specific calibration and QCDomain-specific calibration and QC Data reprocessing and conversionsData reprocessing and conversions Non-automatable data sets (time series)Non-automatable data sets (time series) Custom views unique to domain or mediaCustom views unique to domain or media

SSDS is access point / service providerSSDS is access point / service provider Maintain focus on core services, interfacesMaintain focus on core services, interfaces

21 Nov 2002

19

Development StatusDevelopment Status

First deployed for MOOS Test MooringFirst deployed for MOOS Test Mooring 10/1: Training with development tools10/1: Training with development tools 10/7: First actual code written10/7: First actual code written 11/8: First end to end test (in use ever since)11/8: First end to end test (in use ever since) 11/19: Demonstrated Java GUI for data access11/19: Demonstrated Java GUI for data access 12/3: Data is live from deployed mooring12/3: Data is live from deployed mooring

These are prototype solutions (first round)These are prototype solutions (first round) Planning for AUV CTD data managementPlanning for AUV CTD data management

21 Nov 2002

20

SummarySummary

We designed and built a flexible, dynamic We designed and built a flexible, dynamic data system with an open architecture.data system with an open architecture. Metadata critical to observatory and Metadata critical to observatory and

instrument operation.instrument operation. Standard interfaces enforce modularity. Standard interfaces enforce modularity. A layered metadata model with generic A layered metadata model with generic

concepts provides multiple data access concepts provides multiple data access paths. paths.

Iterative development processes support fast Iterative development processes support fast product deployment and improvements.product deployment and improvements.

21 Nov 2002

21

AcknowledgementsAcknowledgements

Monterey Bay Aquarium Research Monterey Bay Aquarium Research InstituteInstitute David and Lucile Packard FoundationDavid and Lucile Packard Foundation

The SSDS Team: The SSDS Team: Kevin Gomes, John Graybeal, Mike McCann, Kevin Gomes, John Graybeal, Mike McCann,

Brian Schlining, Rich Schramm, Dan WilkinBrian Schlining, Rich Schramm, Dan Wilkin The ISI Team:The ISI Team:

Led by Duane Edgington and Tom O’ReillyLed by Duane Edgington and Tom O’Reilly All our committed and helpful usersAll our committed and helpful users

21 Nov 2002

22

ContactsContacts Shore Side Data System:Shore Side Data System:

John Graybeal, IAG LeadJohn Graybeal, IAG Lead831-775-1956831-775-1956 graybeal@mbari.orggraybeal@mbari.org

Video Annotation and Reference System (VARS):Video Annotation and Reference System (VARS): Dan Wilkin, Lead DeveloperDan Wilkin, Lead Developer

831-775-1865831-775-1865 wilkin@mbari.orgwilkin@mbari.org Samples Database InterfaceSamples Database Interface

Susan Von Thun, Samples CoordinatorSusan Von Thun, Samples Coordinator831-775-2006831-775-2006 svonthun@mbari.orgsvonthun@mbari.org

Cruise (Expedition), SSDS, and ROV 3D Cruise (Expedition), SSDS, and ROV 3D InterfacesInterfaces Mike McCann, Lead DeveloperMike McCann, Lead Developer

831-775-1769831-775-1769 mccann@mbari.orgmccann@mbari.org