automation using python for wvdot gis data management

Automation Using Python for WVDOT GIS Data Management

Yueming Wu, Ph.D., GISPGIS Manager

May 9, 2012

Agenda

IntroductionWVDOT Geospatial Data RepositoryAutomation ImplementationKey Issues & SolutionsFuture WorkQuestions & Answers

Introduction

HistoryHistorically the West Virginia Department of Transportation (WVDOT) has been utilizing geospatial (GIS, Remote Sensing, & GPS) technology for planning, analysis, and mapping purposes ever since the technology became available.

In 2005 the WVDOT GIS Section was created to meet the growing needs for geospatial data and services in transportation.

In 2007 the GIS Section was renamed as the GIS Unit and merged with the Highway Data Services (HDS) Unit, a program responsible for processing changes in roadway information, and the update and maintenance of the Roadway Inventory Log, to form the current Geospatial Transportation Information (GTI) Section.

- Roadway Inventory Log

-Roadway Statistics

-Public Certified Mileage

- Geospatial Data Management

-Geospatial Services

Geospatial Transportation Information (GTI) Section

Program Planning & Administration Division

Highway Data Services UnitGIS Unit

GTI Mission & Vision Statements

Mission StatementTo improve customer service to the citizens of West Virginia by supplying the latest geospatial transportation information.

Vision StatementThe West Virginia Department of Transportation Geospatial Transportation Information (GTI) Section is committed to managing an enterprise Geospatial Information System that will improve the productivity of the West Virginia Department of Transportation, meet growing customer needs, and play a leading role in West Virginia’s geospatial community.

GIS Implementation

Esri Shop Enterprise License Agreement (ELA) Esri Technology

Staff 7 GIS Professionals 2 Cartographers 1 GIS Coordinator at District 7

Enterprise GIS WVDOT Enterprise GIS Roadmap Enterprise Resource Planning (ERP)/Asset Management

System

Major GIS Projects in 2012 Enterprise GIS Implementation

New Transportation GIS Data Model District Level GIS Implementation Workflow Manager

Transportation GIS Data Collection(MS4, Railroad Crossings, Landslides, Outdoor Advertising, Right-of-Way, Trails, Meta Data, etc.) Highway Map Reproduction Web Mapping Applications (STIP, Park & Ride, Functional Classifications, etc.) Mobile GIS Applications (MS4, etc.) 3D GIS Application Development GIS Portal Upgrade ProjectWise Implementation ERP/Asset Management System Implementation

WVDOT Geospatial Data Repository

Export

ReprojectUpload

Update

Store

Download &Add

Metadata

Version - Edit GTI_EDITING

TRANSPORTATIONGIS DATA

• Commissioner Orders• Paper Maps• Field Data• Consultant Reports• Etc.

GTI_PUB_WM

GTI_PUB_UTM

UploadREFERENCE

GIS DATA • WV GIS Tech Center• WVGES• USGS• Esri• Etc.

IMAGERY• Aerial Photos• Shade Relief• Hillshade• Topo• Etc.

Editing

Publication

Source Data

Shapefiles

Imagery GIS Services

Shapefiles for

Download

Store

Download & Add

Metadata

Version - Edit Update

Geospatial Analysis

Map Production

WebApps

Reproject

Export

WVDOT Geospatial Data

Repository

Automation Implementation

Automation ImplementationFour Automated Data Management Operations

GTI_EDITING GDB GTI_PUB_UTM GDB Shapefiles GTI_PUB_UTM GDB GTI_PUB_UTM GDB GTI_PUB_WM GDB GTI_PUB_UTM GDB Shapefiles for Publications

GP1

GP2

GPn

GP …

Data Model

ArcGIS Model Builder

Python Script

Pythonwin

Task

Windows Scheduler

Export Deploy

Improve

Key Issues & Solutions

Database Lock

When a GDB is updated, it’s been accessed by clients (ArcGIS Desktop, ArcGIS Server, etc.). Database locks are created accordingly.

#Remove Database Locks on GTI_PUB_WMImport pyodbccnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=dotarcgis02;DATABASE=GTI_PUB_UTM;UID=xxxxx;PWD=xxx')cursor = cnxn.cursor()cursor.execute("delete FROM [GTI_PUB_UTM].[dbo].[SDE_process_information]" )cnxn.commit()

Scheduled Task

After a Python script passes test, use Windows Scheduler to create a task so the script would run on a regular basis. Make the task independent to individual’s login

Email Notification?

If a task fails, notify the data repository manager by email

#Emailimport smtplib, MIMETexttry:

…except:

sender = 'admin@wvdohwebserver.com‘receivers = ['wuyueming@gmail.com', 'wu_yueming@yahoo.com']msg = MIMEText("The GTI_PUB_UTM_2_Web_Shapefiles script failed. Please check!")

msg['Subject'] = 'Python Script Failed' msg['From'] = "Admin@WVDOHGISWebServer.com" msg['Reply-to'] = "GTI Core Team" msg['To'] = "Admin@WVDOHGISWebServer.com" smtpObj = smtplib.SMTP('localhost') smtpObj.sendmail(sender, receivers, msg.as_string())

or Log

After a task is created, use a log file to track. #LogImport os, sys, timerootFolder = r"C:\Geoprocessing Scripts\\" # This is where the log file is locatedtext_file = open(rootFolder + "PScriptLogFile.txt", "a")text_file.write("\n\n")text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py began at " + str(time.strftime('%X %x %Z')) + ".\n\n")try:

… text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py successfully ended at " +

str(time.strftime('%X %x %Z')) + ".")except: text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py failed at " + str(time.strftime('%X %x %Z')) + ".") text_file.close()

Future Work

Export

ReprojectUpload

Update

Store

Download &Add

Metadata

Version - Edit GTI_EDITING

TRANSPORTATIONGIS DATA

• Commissioner Orders• Paper Maps• Field Data• Consultant Reports• Etc.

GTI_PUB_WM

GTI_PUB_UTM

UploadREFERENCE

GIS DATA • WV GIS Tech Center• WVGES• USGS• Esri• Etc.

IMAGERY• Aerial Photos• Shade Relief• Hillshade• Topo• Etc.

Editing

Publication

Source Data

Shapefiles

Imagery GIS Services

Shapefiles for

Download

Store

Download &Add

Metadata

Version - Edit Update

Geospatial Analysis

Map Production

WebApps

Reproject

Export

WVDOT Geospatial Data

Repository

Replicate

ReprojectUpload

UpdateGTI_EDITING

GTI_PUB_WM

GTI_PUB_UTM

Upload

Editing

Publication

Shapefiles

Update

Reproject

Headquarters

Replicate

Districts 1- 10

GTI_PUB_UTMFGDB

Publication

Replicate

WVDOT Geospatial Data

Repository

Imagery

Yueming WuGIS Manager

WVDOT(304) 558-7437

yueming.wu@wv.gov