data sharing across agencies: a guide to getting on the same page bruce schmidt pacific states...

Data Sharing Across Agencies:A guide to getting on the same page

Bruce SchmidtPacific States Marine Fisheries Commission

Presentation to

Oregon Chapter, AFSFebruary, 2010

www.streamnet.org

CrossJurisdictions

Legal: Remand, US v. OR, PST,

etc.

ESARecoveryPlanning

Coordinated Monitoring

PopulationAssessments

(VSP)

Inter-jurisdictionalManagement

Co-management

High Level Indicators,

SOTR

Increasing Wide-scale Programs

BiOpCompliance

NeedShared

Data

Benefits of Sharing Data

Fosters collaboration

Expand scope of analyses (ESU, DPS)

Contributes to multi-agency processes (e.g.,

ESA analyses, recovery planning and

list/delist decisions; co-management; etc.)

Could distribute cost of data collection

2002Council Contract

with SAIC

Talk of a Comprehensive Regional Data Delivery System…

2000 BiOpRPA 198

CBCIS/ NED

2008 BiOpRPA 72

2006Col. Basin

Data CenterFish and Wildlife

Program

So why isn’t there one?NOAA

Guidance

Impediments to Sharing Data

Sampling is focused Do things differently Data often not

consolidated internally Data not on Web Hard for others to understand, use the data Concerns over misuse, first use, etc. Cost

Current Data Sharing Status:

Individual agencies (States, Tribes, Federal) Independent Different mandates Sample different

environments Longstanding

approaches No mandate to share data Little support, capability to share data

Current Data Sharing Status:

Wide scale database projects Focused – specific data types Not comprehensive

Current Data Sharing Status:

Proposed regional data delivery applications Comprehensive Never built, or

not completed Mechanisms to supply

data not in place

Ideal Distributed Network

…Multiple types of data from multiple sources, standardized by data type

Fish Habitat Water

db db db db db db db db db db db db

OutputTool

RegionalData

Access

Standards Standards Standards

Others

Use Regional Data Projects

db

…Multiple types of data from multiple sources, standardized by data type

WaterHabitatFish

db db db db db db

db

db

OutputTool

RegionalData

Access

dd

dbdbdbdbdb

Others

Pacific Northwest Water Quality Data ExchangeStreamNet

But, Current Reality is more like this:

db

Data sources: field offices, research projects, etc.

DatabaseProject

AgencyDatabase

2 6 95 3 18 7 4

2 6 95 3 18 7 4

2 6 95 3 18 7 4

Consolidated v. Individual Offices Approach

Most Expedient

db

…Multiple types of data from multiple sources, standardized by data type

WaterHabitatFish

db db db db db db

db

db

OutputTool

RegionalData

Access

dd

dbdbdbdbdb

Others

The Essential Driver of a Comprehensive Data Delivery System

db

…Multiple types of data from multiple sources, standardized by data type

WaterHabitatFish

db db db db db db

db

db

OutputTool

RegionalData

Access

dd

dbdbdbdbdb

Data flow can be automated

Others

Automation is the key!

Efficiency, Speed

Accuracy

Automatic data updates

Canned products

Translation to regional format

Essential for any data portal technology

The ultimate endpoint??

…Multiple types of data from multiple sources, standardized by data type

Water Habitat Fish

db db db db db db db db db db db db

Others

Considerations for Regional Data Collection, Sharing and Exchange

‘The Data Sharing Guide’

A White Paper to:

Outline steps

Avoid overlooking critical steps

Describe roles entities play

Break it down: it’s not really that

difficult

ftp://ftp.streamnet.org/pub/streamnet/projman_files/Data_Sharing_Guide_2009-06-01.pdf

How Does the Data Sharing Guide Help?

Organize discussions Consider ALL

components Provide a blueprint Identify needed

support

Assure that when a system is built,the data are there to deliver!

The Data Sharing Guide addresses data flow

from field to highest reporting need

Non prescriptive

Any data type

Any agency

Non-technical

Primary focus is on infrastructure and

processes to assure data accessibility

What’s Needed?

1. Uninterrupted data flow, source to output

Data Flow

Field data creation (local office/project)

Regional data application

Regional standards

Post metadata on the Web

Describe full data set (metadata)

Agency data QA / QC

Agency database system

Local use of data

Describe the data (metadata)

Quality Assurance / Quality Control

Input to electronic form

GOAL

Steps in Data Flow

Post data on the Web

Data interoperability

Data Flow

Field data creation (local office/project)

Regional data application

Post data on the Web

Post metadata on the Web

Describe full data set (metadata)

Agency data QA / QC

Agency database system

Local use of data

Describe the data (metadata)

Quality Assurance / Quality Control

Input to electronic form

GOAL

Steps in Data Flow

Regional standards

Data interoperability

Data Flow

Field data creation (local office/project)

Regional data application

Post data on the Web

Post metadata on the Web

Describe full data set (metadata)

Agency data QA / QC

Agency database system

Local use of data

Describe the data (metadata)

Quality Assurance / Quality Control

Input to electronic form

GOAL

Steps in Data Flow

Regional standards

Data interoperabilityAny missed step can prevent data flow!

Significant effort to bridge the gaps!

What’s Needed?

1. Uninterrupted data flow, source to output

2. Data validated by agency

What’s Needed?

1. Uninterrupted data flow, source to output

2. Data validated by agency

3. Descriptive information (metadata)

X . Y .238 51375 28265 44

What’s Needed?

1. Uninterrupted data flow, source to output

2. Data validated by agency

3. Descriptive information (metadata)

4. Data and metadata on the Internet

What’s Needed?

1. Uninterrupted data flow, source to output

2. Data validated by agency

3. Descriptive information (metadata)

4. Data and metadata on the Internet

XMLSQL

DatabaseSpreadsheet

ASCII

Word Processing

PDF

What’s Needed?

1. Uninterrupted data flow, source to output

2. Data validated by agency

3. Descriptive information (metadata)

4. Data and metadata on the Internet

5. Data able to roll together

Ideal: standard data collection, definition, codes

Minimum: Data translate to a standard

All of this is needed for seamless data

delivery to ANY regional tool

SamplingCrews

SamplingAgencies

FundingEntities

RegionalData Users

DatabaseProjects

PolicyMakers

RolesCreate the dataData EntryQADescribe the dataMaintain the data

Establish proceduresSet standards, codesAgency data systemsPost data / metadataBiological & data mgt. responsibilities

Negotiate key issues Metrics Methods Data dispositionContract languageSupport data automation

Set prioritiesPolicies to remove obstacles to data sharingAgency support

Technical assistanceSystem developmentData tasks for agenciesPost data & metadataFeed regional data tools

The Data Sharing Guide Will Help To:

Inform development of agency data systems

Inform development of a regional data system

Avoid skipping essential components

Get us started

Questions?

Funded by: Through:

Fish and Wildlife Program

Administered by:

www.streamnet.org

Photographs courtesy of CalFish