Enterprise ETL & the Home Again Enterprise ETL & the Home Again Web Mapping Application: Columbus, OhioWeb Mapping Application: Columbus, Ohio

2009 Ohio GIS Conference

September 16-18, 2009

Crowne Plaza North Hotel

Columbus, Ohio

2009 Ohio GIS Conference

September 16-18, 2009

Crowne Plaza North Hotel

Columbus, Ohio

Erick LobaoGIS Project ManagerStantec

Robert ParsonsSoftware EngineerCity of Columbus,Dept. of Technology

Sonia KrammesGIS AnalystCity of Columbus,Dept. of Technology

Nick SoltesGIS AnalystStantec

Overview

• Home Again

• Methodology

• Technical Issues

• Web Application

• Return On Investment

Home Again• Home Again is a Public – Private partnership put forth

by the Mayor to revitalize areas of Columbus by purchasing, renovating, selling properties, and demolishing vacant homes.

• The Department of Development maintains data using several applications and creates reports from these data sources to manage the Home Again initiative.

ColumbusColumbus

Project Goals• Combine data stored in different file formats with data

models / table structures.

• Recreate and automate reports that were manually produced by a former employee.

• Display Home Again point information on a web map and provide a basic level of spatial analysis.

Workflow Diagram

Home AgainArcSDE

GeodatabaseDoT

Columbus Housing

Partnership(Excel

Spreadsheet)

Extract Transform & Load Process

(Store Data in One Table)

Affordable Housing Trust

(Excel Spreadsheet)

City Attorney’s Office

(Unknown)

Vacant Housing Database

(DoT – Oracle)

Join Process Attach Table Information to

Repository Address Points

ArcGIS Server Web Service

(Data Viewable in Web-based Application)

Land Bank Database

(Redevelopment – Approach)

Building Permits(Development -

Accela)

Data Sources

Description Owner Format Frequency

Vacant Housing Application (V.H.A.) Development / Code Enforcement Oracle Table Daily

Land Bank Land Bank Approach Table As Needed

Columbus Housing Partnership CHP Excel Worksheet Monthly

Affordable Housing Trust AHT Excel Worksheet Monthly

Building Permits, Private Demolition Permits

Department of Development Acella (Oracle Table) Daily

Public Demolitions Microsoft Access As Needed

RESPEC Department of Development Microsoft Access Daily

City Attorney's Foreclosure Information

City Attorney's Office Unknown As Needed

This is a Job for Model Builder!• Model Builder allows you to string together ArcGIS

processes to automate multi-step operations.

• Models can be exported to a script and scheduled to run as a scheduled task.

Model Builder scripts can be extended with Java, Python or VBA code if needed.

GIS Workflow

Home AgainArcSDE

GeodatabaseDoT

Columbus Housing

Partnership(Excel

Spreadsheet)

Extract Transform & Load Process

(Store Data in One Table)

Affordable Housing Trust

(Excel Spreadsheet)

City Attorney’s Office

(Unknown)

Vacant Housing Database

(DoT – Oracle)

Join Process Attach Table Information to

Repository Address Points

ArcGIS Server Web Service

(Data Viewable in Web-based Application)

Land Bank Database

(Redevelopment – Approach)

Building Permits(Development -

Accela)

Model goes here

And here

Why Model Builder?• Easy to use.

• Basic understanding of available geoprocessing tools.

• Programming language skills not required.• Future updates, revisions, changes can be made by GIS staff.

• Models are re-usable / speeds up repetitive tasks.

• Execute tasks in a defined order.

• Open a single file to perform a total of 31 geoprocessing operations.

Under The Hood• Models developed with one or two processes at a time.

• Allows you to debug / tune model properly

• Export, format, combine data from multiple sources.

• Join tables based on Parcel ID• Assign spatial information to non-spatial data

• Export to python script for scheduling processes using MS Windows Task Scheduler

Adding Location to Table• Issue: Data stored in a table without spatial reference.

• Address information and “Home Again” attributes stored in a table.

• Solution 1: Create an Oracle table view to join staging table to City’s Master Address Point Layer.• Provides a “Live-View” of final point layer.

• Significant amount of lag time when viewing this layer.

• Solution 2: Use model builder to extract Home Again Points utilizing the coordinates of the Address Point Layer.• Produces a “static” file each time model is executed.

Adding Location to Table

• Join staging table to address points based on Parcel ID.

• Update staging table with (X,Y) from address points.

• Create event feature from staging table.

Adding Location to Table

• Join stage table to address points where Parcel ID = Parcel ID

Adding Location to Table

• Calculate X, Y coordinates within stage table.

Adding Location to Table

• Create event feature class from table.

• Remove any <NULL> X,Y points

Good… But Not Perfect• Adds Complexity.

• Model requires ArcMap to call ArcGIS functions.• Limits use to machines with ArcMap installed.

• ArcGIS file “locking issues” (Not an issue in Python)

• Some operations in Model Builder were buggy.

• Model Builder can do a lot, but at what point do you make the jump to something else?

Web Application

Monthly Reports

ROI: Process• Who’s available to maintain and update ETL

procedures and what’s their skill level?• Leverage existing staff, software, and experience.

• Leverage existing software licenses (ESRI, Crystal Reports).

• Modular approach spreads responsibility.

• Time = $• Repeatable and Automated ETL

ROI: Home Again• Web Application provides GIS capabilities to non GIS

savvy users.• Allows GIS staff to work on higher level tasks.

• Provides spatial analysis capabilities to City leadership.

• Spatially enabling data.• Getting more use / functionality from existing data.

• Development staff can examine relationships between multiple data sources.

• Informs City leadership and Home Again program staff.

A More Permanent Solution• Streamline ETL procedures by improving data

capture and storage.

• Ex. use common field names & data types

• Helps with report generation and data aggregation.

• Web site with forms to enforce data standards.

• Provide flexible reporting.

Questions & Comments

Erick LobaoGIS Project ManagerStantecErick.Lobao@stantec.com(614) 844-4011

Robert ParsonsSoftware EngineerCity of Columbus,Dept. of TechnologyRGParsons@columbus.gov(614) 645-5711

Sonia KrammesGIS AnalystCity of Columbus,Dept. of TechnologySRKrammes@columbus.gov(614) 645-5392

Nick SoltesGIS AnalystStantecNSoltes@stantec.com(614) 486-4383