Evolution of a Clearinghouse:Evolution of a Clearinghouse:From FTP to Map ServicesFrom FTP to Map Services

ESRI International Users Conference, August 10th, 2006ESRI International Users Conference, August 10th, 2006

Maurie Kelly, Ryan Baxter, David WalrathMaurie Kelly, Ryan Baxter, David WalrathPennsylvania Spatial Data AccessPennsylvania Spatial Data Access

www.pasda.psu.eduwww.pasda.psu.edu

In this presentation….In this presentation….

• Historical background of clearinghouse

•Past and current challenges and initiatives

•Related projects and experience

FTP&

Metadata

Relational DatabaseDevelopment

Map Services&

Customization Tools

www.pasda.psu.edu

PASDA …PASDA …

• Is the official public geospatial data clearinghouse for Pennsylvania

PASDA is the Commonwealth’s node on the NSDI, participates in GOS, the Geography

Network, and the NBII

Provides widespread sharing of

Geospatial data

Eliminates thecreation of

redundant data sets

Serves as a resourcefor locating datathroughout PA

PASDA …

•Established in 1996 to serve PA DEP data (35 data sets) via FTP

• Expanded in 1999 to officially serve as the public geospatial data clearinghouse for PA

A collaborative effort of:• PA Geospatial Technologies Office•Penn State Institutes of the Environment•State agencies, programs such as PAMAP, and data partners throughout the region

PASDA…

•Provides access to about 50,000 data files

•Celebrates its 10th anniversary this year and is one of the US’ largest and most established clearinghouses

• Offices are located on PSU UP Campus

PASDAPASDA

• Our services include:

PASDA…

Data Storage and Access

Data &Metad

ata

PASDA

Users

Outreach

•PASDA info.•GIS tutorials

Metadata

Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke

Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke

Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke

Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke

Planar Distance Units:meters Geodetic Model: Horizontal Datum Name:North American Datum of 1927 Ellipsoid Name:Clarke

Training & Assistance

Benefit to Providers:-Mechanism for data distribution-Publicity-Provide service to PA

Benefit to Users:-Can access & use valuable data-Avoid duplication of effort-FREE

How PASDA WorksHow PASDA Works

Data Partners-Data Collection-Mapping-Documentation

PASDA Staff -Storage -Management -Assistance

Distribution Via the Web -Retrieval by user

Time Period of Content:

Time Period Information:

Single Date/Time:

Calendar Date:1994

Currentness Reference:publication date

Status:

Progress:Complete

Maintenance and Update Frequency:None planned

Spatial Domain:

Bounding Coordinates:

West Bounding Coordinate:-74.739

East Bounding Coordinate:-73.652

North Bounding Coordinate:45.156

South Bounding Coordinate:44.496

Keywords:

Theme:

Theme Keyword Thesaurus:None

Theme Keyword:RF3 alpha Hydrography River Reach streams stream network rivers

hydrography ArcInfo navigation drainage

Place:

Place Keyword Thesaurus:Geographic Names Information System

Place Keyword:New York NY New York English-Salmon

Access Constraints:none

Use Constraints:none

Point of Contact:

Contact Information:

Contact Organization Primary:

Contact Organization:U.S. Environmental Protection Agency

Contact Person:Dan Parker

Contact Address:

Address Type:mailing and physical address

Address:401 M Street SW (4503F)

City:Washington

State or Province:DC

Postal Code:20460

Country:United States

Contact Voice Telephone:1-800-424-9067

Data Set Credit:McKay, Lucinda; Sue Hanson; Robert Horn; Richard Dulaney; Alan Cahoon; Mark

Olsen; and Thomas Dewald, 1994. The U.S. EPA Reach File Version 3.0 Alpha Release (RF3-Alpha)

Technical Reference. U.S. Environmental Protection Agency, Washington, DC

PASDA:Data Cataloging

Originator:U.S. Environmental Protection Agency/Office of Water/OST Basins

Originator:U.S. Environmental Protection Agency/Office of Water/OST Basins

Title:USEPA/OW River Reach File 3 (RF3) Alpha for CONUS, Hawaii, Puerto Rico, and the U.S. Virgin Islands

Title:USEPA/OW River Reach File 3 (RF3) Alpha for CONUS, Hawaii, Puerto Rico, and the U.S. Virgin Islands

Spatial Domain:

Bounding Coordinates:

West Bounding Coordinate:-74.739

East Bounding Coordinate:-73.652

Spatial Domain:

Bounding Coordinates:

West Bounding Coordinate:-74.739

East Bounding Coordinate:-73.652

Calendar Date:1994

Indirect Spatial Reference:English-Salmon - New York

Direct Spatial Reference Method:Vector

Description:

Abstract:The U.S. Environmental Protection Agency's (EPA) Reach Files are a series of hydrographic databases of the surface waters of the continental United States …

Description:

Abstract:The U.S. Environmental Protection Agency's (EPA) Reach Files are a series of hydrographic databases of the surface waters of the continental United States …

Planar Distance Units:meters

Geodetic Model:

Horizontal Datum Name:North American

Datum of 1927

Ellipsoid Name:Clarke 1866

Planar Distance Units:meters

Geodetic Model:

Horizontal Datum Name:North American

Datum of 1927

Ellipsoid Name:Clarke 1866

Metadata…

Info about the dataInfo about the data

Types of data on PASDA

•Vector and Raster Data•DEMs• DLGs• DRGs• DOQQs• PAMAP •River Conservation Plans•Census Data•Land Use/Land Cover•Wetlands•Historic Markers•Fish Species

• Data Storage

• 6.5 Terabytes of data

• 3 Terabytes in database for interactive mapping/interactive download

• Access (during last 12 months)

• 11.7 million hits (from 253,000 users)

• 631,000 data downloads requests

• 3.9 Terabytes of data downloaded

• 828,000 online maps generated via web GIS

PASDA Statistics – End of 2005

PASDA Statistics

PASDA StatisticsPASDA Statistics

To whom are we providing assistance?To whom are we providing assistance?

First Challenge—Expand Services and Capability beyond FTP

•#1 Challenge for any clearinghouse—Technology

•Technology changes rapidly both for clearinghouse and for users

•Changes in technology mean changes in hardware, software, and architecture of clearinghouse as well as changes in personnel and training needs.

•Requires regular planning, benchmarking, and vision

Needs Analysis

•1999 PASDA and ATS of Lancaster undertook a needs analysis to define user requirements for role as official public geospatial data clearinghouse for PA

•User requirements included: customization options, web mapping applications, tutorials and guides, more data!!

Impact on Clearinghouse

•Increased training needs and knowledge base for staff

•Increased technology, software, and hardware needs

•More users

Results of Needs Analysis Initiative

Strategic Planning effort undertaken and implemented in 2000 facilitated by Avencia Inc. of Philadelphia.

By 2001, the following were complete

•Development of first relational database using Db2 & SDE

•Pennsylvania Atlas ArcIMS application

•First iteration of Data Wizard customization reprojector/clipping tool

By 2002…

•Statewide DOQQ mosaic in database

•Clipper/reprojector in regular use

•Pennsylvania Atlas ArcIMS application clipping/reprojecting tool developed

By 2003 Technology Changes on the horizon…..

By 2003 changes in technology were on the horizon…

Mid 2003 User requirements session held. Examined existing tools, data, and interface. Acquired input on changes and expectations of users—

•Clearinghouse needed integrated web interface

•Clearinghouse needed streamlined metadata

•Clearinghouse needed integrated option for accessing imagery

•Website and html needed to be streamlined

•Search mechanisms needed to be integrated into single environment

•Post user session—staff held strategic planning session and developed new multiyear plan to implement additional enhancements to clearinghouse.

Challenge….

How to make all of the options accessible?

Solution 1: Approach had been to create new tools for each need Approach had been to add links and content to site

Result 1:Confusing WebsiteCountless links & access points

Challenge Part 2….

How to make all of the options accessible?

Solution 2:

Our approach this past year

Complete redesign of the clearinghouse…search interface, metadata

architecture, data wizard, integration of applications and map

Services into search interface.

Result 2:Simplified WebsiteSingle, integrated access point to all data types & services

Current Architecture…

•“Data cart” option added to Data Wizard

•Users can search for, select, add data to cart, reproject/clip data, and zip all the data in their cart together into one downloadable file.

•Imagery viewer and download utility created for all imagery in database

ArcIMSMapServices

ArcIMSMapServices

NSDI – GOSHarvesting

NSDI – GOSHarvesting

GoogleEarth

GoogleEarth

Clip &Reproject

Clip &Reproject

WMSMapServices

WMSMapServices

WebGISApplications

WebGISApplicationsFTP

.ZIP FilesFTP

.ZIP Files

Current Architecture…

How did we accomplish this?

One central database table

• One record for each ‘dataset’ in the clearinghouse

• Basic metadata elements (title, keywords, date…)

• Links to FTP files

• Links to SDE database feature classes

• Links to XML metadata files

• Links to WebGIS application URLs

• Links to ArcIMS & WMS MapServices

Current Architecture…

Central table drives the Website and integrates all services

Website

Central Table

xml

xml

xml

FTP.zip

SDE

WebGIS

MapServices

ArcIMS -- WMS --Google EarthNSDI -- Geospatial One-Stop

Metadata

DirectDownload

Clip, Reproject& Download

Search by keyword, county, quadrangle, etc.Download .zip files

FTP

Data Access ToolsData Access Tools

DOQQs

View aerial photos; zoom to an addressClick on a quarter-quadrangle to download it

PA Atlas

View & Interact with data on a mapClip, reproject & download

Census 2000

Map & download census 2000 dataAll SF1 & SF3 census variables for PA

Live Internet Data

PennCAT - Geography NetworkCatalog of MapServices

Data Access Wizard

Unified interface for all dataAdd Data to your ‘Data Cart’Clip, reproject and download multiple datasets at one time

View XML metadataView XML metadata

Download via FTPDownload via FTP

Link to separateapplication

Link to separateapplication

Compile datasets forclipping & reprojecting

Compile datasets forclipping & reprojecting

Dynamic ContentDynamic Content

Specialized ApplicationsSpecialized Applications

Imagery Viewer and Download Utility

• 4 Terabytes of imagery including aerial photos DRGs, DOQQs• 5 unique tiling schemes• Multiple years of data—spans period from 1993-2005• All data can be queried via single interface• Search by address,quadrangle, county• Select image and provide with download options http ftp mr. Sid or tiff

Specialized Applications:Imagery ViewerSpecialized Applications:Imagery Viewer

Specialized Applications:Imagery ViewerSpecialized Applications:Imagery Viewer

Specialized Applications:National Weather Service Specialized Applications:National Weather Service Forecast Map ServicesForecast Map Services

Near real time data from NWS including: Apparent Temperature, Dew Point Temperature,Maximum Temperature, Minimum Temperature, Precipitation Amount, Probability of Precipitation, Relative

Humidity, Sky Cover, Snow Amount, Temperature, Wave Height, Wind Direction, and Wind Speed.

Specialized Applications: Tropical Depression IvanSpecialized Applications: Tropical Depression Ivan

• The Ivan viewer was developed to provide information about flood damage following the storm to the public, emergency managers,And other stakeholders in the Commonwealth.

• Ivan includes pre and post flood aerial photos, pictures taken at ground level of flood damage, and supporting framework data.

Ivan Imagery Viewer InterfaceIvan Imagery Viewer Interface

City Island after the flood…City Island after the flood…

Pre-flood imagery of City IslandPre-flood imagery of City Island

Flood damage—Photo taken on site and contributed to Ivan viewer Flood damage—Photo taken on site and contributed to Ivan viewer by citizenby citizen

Pennsylvania Initiatives that use the clearinghouse…Pennsylvania Initiatives that use the clearinghouse…

National Information Infrastructure Initiatives…National Information Infrastructure Initiatives…

ConclusionConclusion

•Clearinghouses have evolved from simple FTPsites to complex information infrastructures•Clearinghouses serve multiple purposes—access to data•Access to services, access to applications, metadata development, and outreach.•PASDA has undergone regular strategic planning to develop new initiatives and to benchmark progress.PASDA has transitioned from a traditional structure to one thatencompasses data, services, and tools for its users.•Map services and applications are now the hallmark of theclearinghouse•Free access to data supports businesses, non-profits, and otherprojects and programs throughout the state and nation.

Questions & CommentsQuestions & CommentsESRI International User Conference, August 10th, 2006ESRI International User Conference, August 10th, 2006

Maurie Kelly, Ryan Baxter, David WalrathMaurie Kelly, Ryan Baxter, David WalrathPennsylvania Spatial Data AccessPennsylvania Spatial Data Access

www.pasda.psu.eduwww.pasda.psu.edu

Please contact us at pasda@psu.edu if you haveany questions or comments.

More information is available athttp://www.pasda.psu.edu/