Maximizing GIS Performance in Your Sleep!

Chris Cushenbery – SSP Innovations

Arnisa Davis – Memphis Light, Gas and Water (MLGW)

Steve Veltman – Memphis Light, Gas and Water (MLGW)

EGUG 2011 1

EGUG 2011 2

Overview

• Utilities are one of the most intense consumers of ESRI geodatabases:– Performing large numbers of edits – Long transactions where ArcSDE versions may be

outstanding for weeks or even months

• Regular system maintenance & performance optimization are key to ensuring GIS providing maximum value to your organization – SSP Innovations worked with MLGW to develop a

suite of batch applications to handle the most common optimization tasks

EGUG 2011 3

MLGW Background

• Largest three-service public utility in the nation– Located in Memphis, TN

• Population & Square Miles– Memphis (676,640) & (324.24 sq mi)– Shelby County (904,703) & (755 sq mi)

• Customers– Electric (422,844)– Gas (314,615)– Water (255,300)

EGUG 2011 4

Batch Suite Architecture

• MLGW Batch Suite dependencies:– Esri ArcGIS Desktop– Telvent ArcFM – SDE Client

• MLGW Batch Suite is a single exe called BatchMaster.exe:– Executed daily by a windows scheduled task– Runs Sun thru Fri at 11pm and Sat at 7PM– Earlier execution on Saturdays to allow additional

processing time for weekly batch reports

EGUG 2011 5

Batch Suite Configuration

• Management of the batch applications occurs via the BatchAppConfig.xml file:

<application powerSwitch="On"name="AppA" class="ClassA" assembly="E:\Apps\ClassA.dll" runLevel="0"runParam=""

/>– powerSwitch: enables and disables whether an application will execute– name: name associated with the application– class: Fully qualified name of the class within the dynamic link library (dll)– assembly: Full path of the dll in which the application resides– runLevel: A numeric setting for the frequency of which the application will run

(daily, weekly, monthly)– runParam: A numeric setting that further designates when the application will

run (day of week, month)

EGUG 2011 6

Optimization Tasks

• A few MLGW optimization tasks:– Orphaned Version Cleanup – Automated Posting of Configured Versions– Full State Tree Reconciliation– Version Conflict Notification– ArcSDE Compresses– Oracle Indexing & Statistics – CIS Interface– FID Initializer– Batch Reports

EGUG 2011 7

Orphaned Version Cleanup

• Purpose:– Compares all JTX Jobs in the geodatabase (GDB) against the

JTX tables. For any jobs that are “complete”, the batch task deletes the corresponding version in the GDB. For any jobs that do not have a corresponding JTX object hence “orphaned”, the batch task deletes the corresponding version in the GDB.

• Frequency: Nightly• Benefits:

– Versions that are “complete” and/or “orphaned” are deleted to ensure there are not any extra versions in the system which might impact system performance. The lower amount of versions in the system means potentially less records in the delta tables which should result in better performance.

EGUG 2011 8

Automated Posting

• Purpose:– Reconciles and posts configured versions to SDE.Default from

a specific XML file. This is typically used whenever a batch application makes updates to an ArcSDE version to post those changes to SDE.Default. Ensures the version is in sync with SDE.Default before processing is executed.

• Frequency: Nightly• Benefits:

– Automated posting of configured versions helps simplify edit operations, reduces the overhead of version administration, and makes sure all edits performed in the configured versions are automatically posted to SDE.Default.

EGUG 2011 9

Full State Tree Reconcile

• Purpose:– Reconciles the entire ArcSDE state tree from SDE.Default down

to all versions in the most effective order. If any conflicts or errors are found in individual versions they are logged and emailed out to a configured list using the Version Conflict Notification task.

• Frequency: Nightly• Benefits:

– Performance can be affected by the size of the Esri state table which tracks outstanding edits across all versions. By reconciling all posted edits in an optimal order from SDE.Default down into the outstanding versions allows an ArcSDE compress to be more effective in minimizing the size of the Esri state table.

EGUG 2011 10

Version Conflict Notification

• Purpose:– Reports on version conflicts by either posting a report to an

intranet website or emailing a configured list of users with a list of conflict versions to be resolved.

• Frequency: Nightly• Benefits:

– Provides notification by email or intranet website of versions in conflict on a nightly basis. Versions in conflict need to be resolved in a timely manner since these versions are unable to be reconciled. Continuing to ignore versions that are unable to be reconciled will impact the size of the Esri state table thus impacting performance.

EGUG 2011 11

ArcSDE Compress

• Purpose:– Compressing the ArcSDE state tree removes each state that is

not currently referenced by a version and is not the parent of multiple child states. This task takes all posted data that has been reconciled to all versions and moves it from the delta tables into the base tables.

• Frequency: Nightly• Benefits:

– The ArcSDE compress minimizes the size of the state tree, shortening the lineage and improving query performance.

EGUG 2011 12

Oracle Indexing & Statistics

• Purpose:– The batch database indexing and statistics application rebuilds

the indices and updates the statistics for all GIS tables to improve performance.

• Frequency: Nightly / Weekly• Benefits:

– Database management systems (DBMSs) determine the most efficient way to execute the queries sent to them based on database statistics. Therefore, for optimal performance of datasets created within an ArcSDE geodatabase, you need to keep the statistics current by frequently updating.

EGUG 2011 13

CIS Interface

• Purpose:– Handles synching the CIS and GIS data on a nightly basis.

Updates the StreetAddress.LifeSupportIndicator field from CIS, it will set the premise code for any StreetAddress features that are missing this value if it can match to CIS based on address, it will keep the CISPremisePoint feature class updated from CIS (representation of all customer points on the map).

• Frequency: Nightly• Benefits:

– Synchronizing information from the CIS into the GIS enables CIS tools for querying, validation, reporting, and analysis.

EGUG 2011 14

FID Initializer

• Purpose:– FID Sequence values can be automatically generated by the

system. For any configured feature classes, the user can create features in the database and set the FIDSequence value to “99999”. The batch application will process the posted edits looking for any records with FIDSequence=99999.

• Frequency: Nightly• Benefits:

– For the configured feature classes the application assigns the correct next FID to the feature based on the FIS Section & Block and the commodity to ensure data accuracy and consistency of feature numbering.

EGUG 2011 15

Batch Reports

• A few MLGW Batch Reports:– CIS Address Comparison: Compares all street addresses in

GIS to the corresponding CIS and MSS datastores and creates a report of all the differences in GIS. This report is distributed via email. This application is configured to execute last on Saturdays because it is a long-running report. It can be run weekly or shifted to run monthly.

– Annexation Tracker: Compares the annexation polygons for changes since the last run of the AnnexationTracker application and then reports on all Street Addresses that are affected by the changes. The report is distributed via email. This can be run weekly or shifted to run monthly.

EGUG 2011 16

Batch Report Log

• Reporting (Batch.log):– Most processes (FID, SystemReconcile, etc)– RebuildIndecies.bat.log– UpdateStatistics.bat.log– CIS Interface (LifeSupport indicator, Update GIS

features from CIS database)

EGUG 2011 17

Batch Report Log

• Reporting (Batch.log):– Orphaned versions

– FID Updates

– System reconcile

EGUG 2011 18

Summarizing MLGW Benefits

• Provides ability to generate unique FID numbers for features like valves or fittings

• Synchronizes our CIS billing system with our GIS on a nightly basis

• Compresses the ArcSDE state tree– Takes all posted data that has been reconciled to ALL

versions and moves it from the A&D tables into the base tables

EGUG 2011 19

Summarizing MLGW Benefits

• Updates the Oracle statistics for all GIS tables to improve performance

• Provides the ability to run weekly reports over the weekend without human intervention– CIS address differences– New annexations

EGUG 2011 20

Questions

• Thank you for attending• For more information contact:

Chris CushenberySSP Innovations, LLC

chris.cushenbery@sspinnovations.com

Arnisa DavisMLGW

adavis@mlgw.org

Steve VeltmanMLGW

sveltman@mlgw.org