from oracle to the web - automating spatial data updates

From Oracle to the Web - Automating Spatial Data Updates David Smith, FRS Program Manager, EPAAmy Ramsdell, GIS Manager, Blue Raster

April 8

Overview

The Need: Automate spatial data refresh The Solution: Technologies used ArcGIS Server (AGS) map service final output FME workspaces that automate refresh process

File geodatabase (fgdb) creation Metadata refresh

Python shutdown script in workspace FME Server web services used EPA future plans for FME Questions

The Need:

Automate refresh of map service serving spatial data

Operational database system The Facility Registry Service (FRS) is a centrally managed

database that identifies facilities, sites or places subject to environmental regulations or of environmental interest.

Disconnected static file for map service Fgdb faster performance in AGS Lessen activity on operational database

Monthly manual refresh

The Solution:

ESRI ArcGIS Server Final Output

Web server hosting GIS web services

REST endpoint for map servicehttp://geodata.epa.gov/ArcGIS/rest/services/OEI/FRS_INTERESTS/MapServer

Underlying data: file gdb 28 layers (tables in fgdb)

Data Refresh FME Workspace

Oracle Spatial reader

ESRI File Geodatabase writer

Whole bunch of filtering in-between to create separate tables in fgdb Could build the

filtering in Oracle before it reaches FME

Oracle Spatial Type Reader

Required for spatial object: Table, materialized view, or view

Shape field is SDO_GEOMETRYCREATE TYPE sdo_geometry AS OBJECT ( SDO_GTYPE NUMBER, SDO_SRID NUMBER, SDO_POINT SDO_POINT_TYPE, SDO_ELEM_INFO SDO_ELEM_INFO_ARRAY, SDO_ORDINATES SDO_ORDINATE_ARRAY);

USER_SDO_GEOM_METADATA table entry Metadata for spatial tables owned by schema

ESRI File Geodatabase API Writer

Non-ArcObjects access to fgdb

C++ API

Does not require an ArcGIS license on FME Server or Desktop

Works in ArcGIS Server 10x

Python Shutdown Script in Workspace

In Workspace Parameters – Advanced option

Shutdown Script in 3 Steps

1. Package the output zipfgdb(geodatabase, path)

2. Refresh the map service updateArcGISServer(server, services, _ZIP,

_ZIPDESTINATION)

3. Notify dependent subscription workspace NotifySuccess()

Shutdown Script: 1. Package the Output

Zip the file geodatabase

Shutdown Script: 2. Refresh Map Service

1. Stop map service

2. Delete current fgdb

3. Copy to shared network drive

4. Extract new fgdb

5. Start map service

1

2

4

3

5

Refresh Map Service: Stop/Start Services

Stopping and starting AGS map service

Refresh Map Service: AGS Token Access

Generate token for admin access

Refresh Map Service: Replace Fgdb

Delete existing fgdb and extract zip

Shutdown Script: 3. Notify Subscription

FME Server

FME Notification Service - Topics

Create a Success and a Fail notification topic to perform some action based on the data refresh workspace execution status

FME Notification Service - Subscriptions 2 subscribers

Email service to notify when the workspace fails Metadata workspace runs if data refresh workspace completes

successfully

Metadata about Map Service

FGDC – Compliant metadata required in XML format XML file updated for date of fgdb refresh

https://edg.epa.gov/

Subscriber: Metadata Refresh Workspace

EPA Future Plans for FME

Kick off workspace from Oracle instead of manually in FME Server’s Services interface – good for more frequent refresh cycles than monthly using FME’s Job Submitter service http://

servername.com/fmejobsubmitter/Samples/workspacename.fmw?FMEServerHost=http://servername.com&SuccessTopic=InternetFGDB&Token=4fab5b76c6yufed2c4731s3d0387bfb366569949&FailTopic=InternetFGDBFail&UserName=admin&Password=password&opt_showresult=false&opt_servicemode=sync

Possibly enhance FRS Query Tools by adding spatial selections and making more file formats available

Integrate into other EPA program map service refresh flows

Research FME’s ability to translate Resource Description Framework (RDF), a semantic web data format

EPA Facility Registry Services (FRS)

Thank You!

Questions? For more information:

David Smith, EPA Facility Registry Service Smith.DavidG@epamail.epa.gov

@DruidSmith

Judith Doherty, FRS Project Manager, Indus Corporation jdoherty@induscorp.com

Amy Ramsdell, Blue Raster aramsdell@blueraster.com

www.blueraster.com/blog @AmyRams