curing migration flu or: how i learned to stop worrying and love the lgim

Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM Carl Von Stetten GIS Analyst, Central Contra Costa Sanitary District

The Migration Flu comes in cycles

Currently on 3rd mapping system migration FME integral to moving between GIS systems

The long journey…

For decades, CentralSan has been trying to get from hardcopy map books…

Picture courtesy http://esri.com

…to a fully integrated digital environment.

In the old days…

From at least the late ‘70s on, this was our GIS:

CentralSan goes digital!

Transition to digital mapping began in 1989 Platform was Intergraph’s MGE

System running on Bentley MicroStation Ran on UNIX-based CLIX

workstations CAD files with database

attachments for attributes

The flue began with a fever…

MGE was far from perfect UNIX environment: MGE/MicroStation + RIS +

Informix Three different software companies Different release cycles Brittle version compatibility

Windows environment: MGE/MicroStation + RIS + MS SQL Server Same problems, just traded database platforms

…followed by the chills…

Data itself wasn’t always reliable MicroStation file corruption

Entire software product line (Axiom International) developed just to repair corrupt files

Broken database linkages Orphaned database records

Data split into hundreds of tiles/basemaps 739 sewer map tiles / 24 recycled water tiles 137 assessor map base files

But the hardcopy product got better…

The seamless GIS begins

In late ‘90s/early ‘00s began developing web-based GIS application California CAD Solutions (CalCAD) helped deploy

Autodesk MapGuide FME was used to convert Microstation tiles into a

seamless set of Mapguide .SDF files and attribute database tables

Benefits realized

Seamless map environment enabled integration with external systems Computerized Maintenance Management System

(CMMS) data from Sussex/Accela Parcel and permit data from Sungard ERP system Pipeline CCTV inspection data from WinCan

…oh, the aches…

MGE problems a continuous pain in the @#! 1. Run the FME processes 2. Discover corrupted files/broken database links “Holes” in seamless map

3. Repair the files/reenter database data 4. Rerun FME 5. Discover new corrupt files/broken links 6. Rinse and repeat…

Curing the flu

In 2005, chose Intergraph GeoMedia 5.2 with Public Works Manager (PWM) extension to replace MGE

Designed a custom schema to contain all the different feature classes from MGE/MicroStation files

Every element had to have an MGE “linkage” to be able to migrate into GeoMedia

FME to the rescue!

Wrote FME workbenches to prep data for migration: Find MicroStation elements without MGE “linkages” Find elements with incomplete or invalid attributes

If Intergraph hadn’t been contracted to perform the data migration, we would have used FME

Wrote new FME workbenches to convert GeoMedia data to MapGuide / SQL Server for web GIS Still using mostly same FME workbenches 9 years

later

And the flu symptoms return…

Deployed GeoMedia/PWM 5.2 Intergraph wrote significant amount of custom

PWM code to automate workflows Bugs in display and storage of arcs

GeoMedia/PWM 6.0 released shortly after implementation completed API completely changed Rewrite of custom code approx. $50K – no way! Could only use for printing/publishing (fixed arcs

bug)

If only there was a flu shot…

GeoMedia 6.1 released a few years later Another API change Cost to catch up custom code rose to $100K

GeoMedia 5.2 won’t run reliably on Windows 7 Resorted to maintaining Windows XP virtual

machines for compatibility

Another hardcopy improvement though…

Migration time again…

Began looking for a replacement to GeoMedia in 2011

Esri ArcGIS chosen, planning begun Introduced to ArcGIS Local Government

Information Model (LGIM) Standardized data model Supported by many 3rd party systems (CMMS,

CCTV, hydraulic modeling, ERP, etc.)

…and FME rescues us again!

Splitting data migration workloads with Esri Both CentralSan and Esri are using FME

Successfully writing to both Esri file geodatabase

and Microsoft SQL Server enterprise geodatabase formats

About 60% complete with FME workbench development and testing

What does the final cure look like?

Eliminate hardcopy maps entirely Focused applications for each

department/workgroup Mobile solutions for map viewing and field data

collection GIS will be authoritative inventory source for

assets Tight integration with CMMS and CCTV systems

Now for some tips…

Three things we picked up along the way

Tip #1 – Keep DRY

In programming, DRY is a principle of code organization Don’t Repeat Yourself Move code that will repeat over and over again into

a subroutine or stand-alone module Applies equally well to FME If you will reuse sequences of transformers

multiple times, put them in custom transformers Export custom transformers to share and reuse in

different workbenches

Tip #1 – Keep DRY

Tip #2 - SchemaMapper

Replaces multiple AttributeCreator, AttributeMapper, FeatureTypeFilter transformers

Can remap feature types too Uses CSV file, Excel file (.xls/.xlsx), or database

table to store mappings/values Benefits: Reduce transformer count/complexity External table can be edited/maintained by

someone other than FME workbench author

Tip #2 - SchemaMapper

Tip #3 – Version Control

Changes to workbenches can be tracked in FME:

Tip #3 – Version Control

Workbench files are text files, so they can be effectively managed in various version control systems Examples: Git, Mercurial, SVN, Team Foundation

Provides ability to track history, roll back revisions if needed

Using on-premise or hosted version control services provides a backup & recovery solution Examples: Github, BitBucket

Tip #3 – Version Control

Thank You!

Questions?

For more information: Carl Von Stetten (cvonstetten@centralsan.org)

Web: http://centralsan.org Twitter: @cfvonner