curing migration flu or: how i learned to stop worrying and love the lgim
TRANSCRIPT
Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM Carl Von Stetten GIS Analyst, Central Contra Costa Sanitary District
The Migration Flu comes in cycles
Currently on 3rd mapping system migration FME integral to moving between GIS systems
The long journey…
For decades, CentralSan has been trying to get from hardcopy map books…
Picture courtesy http://esri.com
…to a fully integrated digital environment.
In the old days…
From at least the late ‘70s on, this was our GIS:
CentralSan goes digital!
Transition to digital mapping began in 1989 Platform was Intergraph’s MGE
System running on Bentley MicroStation Ran on UNIX-based CLIX
workstations CAD files with database
attachments for attributes
The flue began with a fever…
MGE was far from perfect UNIX environment: MGE/MicroStation + RIS +
Informix Three different software companies Different release cycles Brittle version compatibility
Windows environment: MGE/MicroStation + RIS + MS SQL Server Same problems, just traded database platforms
…followed by the chills…
Data itself wasn’t always reliable MicroStation file corruption
Entire software product line (Axiom International) developed just to repair corrupt files
Broken database linkages Orphaned database records
Data split into hundreds of tiles/basemaps 739 sewer map tiles / 24 recycled water tiles 137 assessor map base files
But the hardcopy product got better…
The seamless GIS begins
In late ‘90s/early ‘00s began developing web-based GIS application California CAD Solutions (CalCAD) helped deploy
Autodesk MapGuide FME was used to convert Microstation tiles into a
seamless set of Mapguide .SDF files and attribute database tables
Benefits realized
Seamless map environment enabled integration with external systems Computerized Maintenance Management System
(CMMS) data from Sussex/Accela Parcel and permit data from Sungard ERP system Pipeline CCTV inspection data from WinCan
…oh, the aches…
MGE problems a continuous pain in the @#! 1. Run the FME processes 2. Discover corrupted files/broken database links “Holes” in seamless map
3. Repair the files/reenter database data 4. Rerun FME 5. Discover new corrupt files/broken links 6. Rinse and repeat…
Curing the flu
In 2005, chose Intergraph GeoMedia 5.2 with Public Works Manager (PWM) extension to replace MGE
Designed a custom schema to contain all the different feature classes from MGE/MicroStation files
Every element had to have an MGE “linkage” to be able to migrate into GeoMedia
FME to the rescue!
Wrote FME workbenches to prep data for migration: Find MicroStation elements without MGE “linkages” Find elements with incomplete or invalid attributes
If Intergraph hadn’t been contracted to perform the data migration, we would have used FME
Wrote new FME workbenches to convert GeoMedia data to MapGuide / SQL Server for web GIS Still using mostly same FME workbenches 9 years
later
And the flu symptoms return…
Deployed GeoMedia/PWM 5.2 Intergraph wrote significant amount of custom
PWM code to automate workflows Bugs in display and storage of arcs
GeoMedia/PWM 6.0 released shortly after implementation completed API completely changed Rewrite of custom code approx. $50K – no way! Could only use for printing/publishing (fixed arcs
bug)
If only there was a flu shot…
GeoMedia 6.1 released a few years later Another API change Cost to catch up custom code rose to $100K
GeoMedia 5.2 won’t run reliably on Windows 7 Resorted to maintaining Windows XP virtual
machines for compatibility
Another hardcopy improvement though…
Migration time again…
Began looking for a replacement to GeoMedia in 2011
Esri ArcGIS chosen, planning begun Introduced to ArcGIS Local Government
Information Model (LGIM) Standardized data model Supported by many 3rd party systems (CMMS,
CCTV, hydraulic modeling, ERP, etc.)
…and FME rescues us again!
Splitting data migration workloads with Esri Both CentralSan and Esri are using FME
Successfully writing to both Esri file geodatabase
and Microsoft SQL Server enterprise geodatabase formats
About 60% complete with FME workbench development and testing
What does the final cure look like?
Eliminate hardcopy maps entirely Focused applications for each
department/workgroup Mobile solutions for map viewing and field data
collection GIS will be authoritative inventory source for
assets Tight integration with CMMS and CCTV systems
Now for some tips…
Three things we picked up along the way
Tip #1 – Keep DRY
In programming, DRY is a principle of code organization Don’t Repeat Yourself Move code that will repeat over and over again into
a subroutine or stand-alone module Applies equally well to FME If you will reuse sequences of transformers
multiple times, put them in custom transformers Export custom transformers to share and reuse in
different workbenches
Tip #1 – Keep DRY
Tip #2 - SchemaMapper
Replaces multiple AttributeCreator, AttributeMapper, FeatureTypeFilter transformers
Can remap feature types too Uses CSV file, Excel file (.xls/.xlsx), or database
table to store mappings/values Benefits: Reduce transformer count/complexity External table can be edited/maintained by
someone other than FME workbench author
Tip #2 - SchemaMapper
Tip #3 – Version Control
Changes to workbenches can be tracked in FME:
Tip #3 – Version Control
Workbench files are text files, so they can be effectively managed in various version control systems Examples: Git, Mercurial, SVN, Team Foundation
Provides ability to track history, roll back revisions if needed
Using on-premise or hosted version control services provides a backup & recovery solution Examples: Github, BitBucket
Tip #3 – Version Control
Thank You!
Questions?
For more information: Carl Von Stetten ([email protected])
Web: http://centralsan.org Twitter: @cfvonner