mining for lost treasure national geospatial data clearinghouse archibald warnock u.s. federal...
TRANSCRIPT
Mining For Lost TreasureMining For Lost Treasure
National Geospatial Data ClearinghouseNational Geospatial Data Clearinghouse
Archibald WarnockArchibald WarnockU.S. Federal Geographic Data CommitteeU.S. Federal Geographic Data Committee
A/WWW EnterprisesA/WWW Enterprises
What is Clearinghouse?What is Clearinghouse?
A distributed service to locate A distributed service to locate geospatial data based on geospatial data based on characteristics expressed in characteristics expressed in metadatametadata
Clearinghouse allows a user to pose a Clearinghouse allows a user to pose a query of all or a portion of the query of all or a portion of the community in a single sessioncommunity in a single session
Like a spatial AltaVista Like a spatial AltaVista
National Geospatial Data National Geospatial Data ClearinghouseClearinghouse
Distributed data producers and Distributed data producers and users.users.
Key components:Key components:– Data documentation (metadata)Data documentation (metadata)– Networking (Internet)Networking (Internet)– Serving, searching, and accessing Serving, searching, and accessing
softwaresoftware Z39.50 Search and Retrieve ProtocolZ39.50 Search and Retrieve Protocol WWW - World Wide WebWWW - World Wide Web
Components of Components of ClearinghouseClearinghouse There are three functional areas There are three functional areas
that interact to create the that interact to create the Clearinghouse:Clearinghouse:– Metadata preparation and indexingMetadata preparation and indexing– Metadata serviceMetadata service– User Access via Gateway formsUser Access via Gateway forms
Clearinghouse MethodClearinghouse Method
Metadatapreparation
Metadatavalidation/
staging
Metadatapublication
Useraccess
Clearinghouse DesignClearinghouse Design
The Clearinghouse in its distributed The Clearinghouse in its distributed form includes a registry of servers, form includes a registry of servers, several WWW-to-Z39.50 gateways, several WWW-to-Z39.50 gateways, and many Z39.50 serversand many Z39.50 servers
A primary goal of Clearinghouse is A primary goal of Clearinghouse is to provide the ability to find spatial to provide the ability to find spatial data throughout the entire data throughout the entire community, not one site at a timecommunity, not one site at a time
Essential ConfigurationEssential Configuration
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
UserUser downloads query downloads query formform
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
User sends query to web serverUser sends query to web server
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
Gateway passes query to Clearinghouse Gateway passes query to Clearinghouse ServersServers
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
Gateway receives and collates “hits”Gateway receives and collates “hits”
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
Client receives results summary as HTMLClient receives results summary as HTML
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
Client can request a specific metadata record Client can request a specific metadata record for viewingfor viewing
FGDCFGDC
Gateways
WebClient
WebClient
NodeNode
NodeNode
NodeNode
NodeNode
Clearinghouse Sites
Node in More DetailNode in More Detail
MetadataIndex/DBZ39.50server
Internet Data
DataData
The most expensive investment for The most expensive investment for an organizationan organization
Created by many different Created by many different organizationsorganizations
To solve many different problemsTo solve many different problems Using many different methods and Using many different methods and
technologiestechnologies
But . . .But . . .
Data are hard to findData are hard to find Data are difficult to accessData are difficult to access Data are hard to integrateData are hard to integrate Data are not currentData are not current Data are undocumentedData are undocumented Data are incompleteData are incomplete
The uses of metadataThe uses of metadata
Provides documentation of existing Provides documentation of existing internal geospatial data resources internal geospatial data resources within an organizationwithin an organization (inventory)(inventory)
Permits structured search and Permits structured search and comparison of held spatial data by comparison of held spatial data by othersothers (advertising)(advertising)
Provides end-users with adequate Provides end-users with adequate information to take the data and use it information to take the data and use it in an appropriate contextin an appropriate context (liability)(liability)
Metadata SolutionsMetadata Solutions
Numerous software solutions Numerous software solutions availableavailable
Commercial and free-wareCommercial and free-ware Standalone, DB-linked, GIS-linkedStandalone, DB-linked, GIS-linked Permit collection and structuring of Permit collection and structuring of
FGDC-compatible metadataFGDC-compatible metadata Present metadata as HTML, XML, or Present metadata as HTML, XML, or
texttext
GILS, Dublin Core and GILS, Dublin Core and OthersOthers Dublin Core is a minimal (15 fields) generic Dublin Core is a minimal (15 fields) generic
metadata scheme for virtually any kind of metadata scheme for virtually any kind of documentdocument
GILS represents a more detailed approach, GILS represents a more detailed approach, including most of DC, providing greater including most of DC, providing greater interoperabilityinteroperability
GILS is less bibliographically oriented than GILS is less bibliographically oriented than (Z39.50) BIB-1(Z39.50) BIB-1
GILS is lightweight compared to GEO (FGDC) GILS is lightweight compared to GEO (FGDC) and EOS/CIP (which have specific functional and EOS/CIP (which have specific functional requirements)requirements)
What Structured Metadata What Structured Metadata Means -1Means -1
GILS - Fewer GILS - Fewer fieldsfieldsMore documentsMore documentsMore metadata More metadata
recordsrecordsSkinnier metadata Skinnier metadata
recordsrecordsEasier abstractionEasier abstraction
FGDC - More FGDC - More fieldsfieldsFewer documentsFewer documentsFewer metadata Fewer metadata
recordsrecordsFatter metadata Fatter metadata
recordsrecordsLess abstractionLess abstraction
GILS is a good, general compromise
What Structured Metadata What Structured Metadata Means - 2Means - 2
A Z39.50 profile as defines a languageA Z39.50 profile as defines a language At some level, Z39.50 is a detailAt some level, Z39.50 is a detail Protocols are about communication, profiles are about Protocols are about communication, profiles are about
abstraction and GILS is about contentabstraction and GILS is about content Z39.50 guarantees that the user’s query can be Z39.50 guarantees that the user’s query can be
unambiguously decoded - no guarantees about contentunambiguously decoded - no guarantees about content We could implement the profile over any protocol - We could implement the profile over any protocol -
http, CORBA, etc.http, CORBA, etc.
Do we have to use Z39.50?Do we have to use Z39.50? No, but the abstraction is requiredNo, but the abstraction is required Z39.50 already includes the abstraction modelZ39.50 already includes the abstraction model
How much metadata is How much metadata is enough?enough? Internal documentation for local use Internal documentation for local use
(local inventory)(local inventory) Basic documentation for discovery of Basic documentation for discovery of
information holdings information holdings (catalog/search)(catalog/search) Detailed documentation to provide Detailed documentation to provide
end-users with adequate information end-users with adequate information for re-use for re-use (asset management)(asset management)
Server SolutionsServer Solutions
Z39.50 Protocol is usedZ39.50 Protocol is used ““GEO” Geospatial Metadata Profile is GEO” Geospatial Metadata Profile is
published for Z39.50 implementors to published for Z39.50 implementors to understand FGDC metadata understand FGDC metadata structuresstructures
Supports search across numeric, text, Supports search across numeric, text, date, and spatial extent and full-textdate, and spatial extent and full-text
Freeware and commercial solutions Freeware and commercial solutions
Gateway in more Gateway in more detaildetail
Nodes
GatewayGatewayWeb
serverinterface
Z39.50clients
Web Gateway Case
Webclient
Webclient
User InterfacesUser Interfaces
HTML-based forms hosted at HTML-based forms hosted at Gateways are the primary access Gateways are the primary access methodmethod
Java map-based interface from Java map-based interface from MEL allows more sophisticated MEL allows more sophisticated searchsearch
Inclusion of search capabilities in Inclusion of search capabilities in GIS client software is possibleGIS client software is possible
Who’s in Clearinghouse?Who’s in Clearinghouse?
109 Nodes (servers) online as of 109 Nodes (servers) online as of 3/1/993/1/99– 28 Federal, national scope28 Federal, national scope– 35 State/University state-wide scope35 State/University state-wide scope– 28 International scope or location28 International scope or location– 18 Local or Regional scope18 Local or Regional scope
US Federal ParticipationUS Federal Participation
NOAA (10)NOAA (10) USGS (6)USGS (6) FEMA (sampler)FEMA (sampler) NRCS climate and NRCS climate and
soilssoils CIESIN/EPACIESIN/EPA CIESIN/NASACIESIN/NASA DOT NTADDOT NTAD
National Park ServiceNational Park Service Army Corps of Army Corps of
EngineersEngineers Tri-Services CenterTri-Services Center National Wetlands National Wetlands
InventoryInventory Census (sampler)Census (sampler) Minerals Minerals
Management ServiceManagement Service
State ParticipationState Participation
New York (2)New York (2) North CarolinaNorth Carolina OklahomaOklahoma KansasKansas TexasTexas Montana (3)Montana (3) VermontVermont PennsylvaniaPennsylvania
West VirginiaWest Virginia WashingtonWashington WisconsinWisconsin Wyoming (2)Wyoming (2) FloridaFlorida AlabamaAlabama New MexicoNew Mexico ArizonaArizona
GeorgiaIllinoisMinnesotaAlaskaCaliforniaDelawareNebraska (2)New Jersey
Regional/Local Regional/Local ParticipationParticipation McKinley Co, NMMcKinley Co, NM City of Santa Fe, NMCity of Santa Fe, NM North Texas GISNorth Texas GIS Research PlanningResearch Planning Sabine R Authority, TXSabine R Authority, TX San Francisco BaySan Francisco Bay S Florida EcosystemS Florida Ecosystem SW Natural ResourcesSW Natural Resources
Olympic Peninsula, WAOlympic Peninsula, WA Greater YellowstoneGreater Yellowstone Helena NFHelena NF Ecological Reserves, KSEcological Reserves, KS MIT/Mass Boston DOQsMIT/Mass Boston DOQs Great Lakes EISGreat Lakes EIS Eastern SierraEastern Sierra
International ParticipationInternational Participation
NOAA/Japan GOINNOAA/Japan GOIN South Africa (2)South Africa (2) ESA AVHRR samplerESA AVHRR sampler GELOS, ItalyGELOS, Italy PAIGH, MexicoPAIGH, Mexico S57 Hydrography, CanadaS57 Hydrography, Canada NRL MELNRL MEL Africa DDSAfrica DDS Inter-American Geospatial Data Inter-American Geospatial Data
NetworkNetwork Hong KongHong Kong CIESIN/USDA Global CIESIN/USDA Global
Environmental ChangeEnvironmental Change Australia (10+)Australia (10+) Costa RicaCosta Rica Caribbean CEPNET, JamaicaCaribbean CEPNET, Jamaica
Planned or Funded NodesPlanned or Funded Nodes
Mt Desert Island, MEMt Desert Island, ME SW Washington COGSW Washington COG NASA GCMDNASA GCMD CODEPLAN, BrazilCODEPLAN, Brazil IowaIowa Missouri Missouri KentuckyKentucky
South DakotaSouth Dakota OregonOregon LouisianaLouisiana OhioOhio Connecticut MAGICConnecticut MAGIC ColoradoColorado NW EcosystemsNW Ecosystems
Clearinghouse provides...Clearinghouse provides...
Discovery Discovery of spatial data of spatial data Distributed Distributed search worldwidesearch worldwide Uniform interfaceUniform interface for spatial datafor spatial data
searchessearches Advertising Advertising for your data holdingsfor your data holdings
For more information:For more information:
Visit the FGDC website: http://www.fgdc.gov
Contact the Clearinghouse Coordinator, Doug Nebert ([email protected]) or Archie Warnock ([email protected])