2013 tips and tricks mashup, from modelbuilder to formal python code, step-by-step by zeke houk

ModelBuilder to Python:

Step-by-Step

Zeke Houk

GIS Developer

ModelBuilder Model:

Delete Features from

simple_conduits:

Pull snapshot from SDE:

Get Rid of Conduit 0:

Dissolve on Conduit Number:

Guido Van Rossum

BDFL

Where did the name come from?

Python script:

Clean up names:

• >>> # Local variables:

• >>> simple_conduits__3_ = R:\\ARG_Projects\\ConduitMapBook\\DissolvedConduits.gdb\\simple_conduits"

• >>> dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

• >>> ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

• >>> simple_conduits__4_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

• >>> N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"

• >>> dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

• >>> simple_conduits__6_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

>>> simple_conduits_3_,

>>> simple_conduits_4_,

>>> simple_Conduits_6_

>>> redundant…

Replace with simple_conduits:

• >>> # Local variables:

• >>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

• >>> dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

• >>> ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

• >>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

• >>> N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"

• >>> dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

• >>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

Global replace simple_conduits

For simple_conduits_3_,

simple_conduits_4_, and

simple_conduits_6_.

Duplicates dropped:

>>> # Local variables:

>>> simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

>>> dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

>>> ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

>>> N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"

>>> dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

Global replace simple_conduits

For simple_conduits_3_,

simple_conduits_4_, and

simple_Conduits_6_.

Now fix dissolved_conduits:

# Local variables:

simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"

Becomes…

# Local variables:

simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

dissolved_conduits__3_ = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

ARG_CONDUIT__3_ = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"

dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

Use File Geodatabase Name:

• # Local variables:

• file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"

• simple_conduits = file_gdb + "simple_conduits"

• dissolved_conduits = file_gdb + "dissolved_conduits"

• N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"

• ARG_CONDUIT = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

Becomes…

• # Local variables:

• simple_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\simple_conduits"

• dissolved_conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\dissolved_conduits"

• ARG_CONDUIT = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

• N0_Zero_Conduits = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\N0_Zero_Conduits"

Use File Geodatabase Name:

file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"

simple_conduits = file_gdb + "simple_conduits"

dissolved_conduits = file_gdb + "dissolved_conduits"

N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"

Use SDE Connection Name:

# Local variables:

file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"

simple_conduits = file_gdb + "simple_conduits"

dissolved_conduits = file_gdb + "dissolved_conduits"

N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"

sde_connection = "Database Connections\\Connection to DWGIS3.sde\\"

ARG_CONDUIT = sde_connection + "ARG.CONDUIT"

Becomes…

# Local variables:

file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"

simple_conduits = file_gdb + "simple_conduits"

dissolved_conduits = file_gdb + "dissolved_conduits"

N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"

ARG_CONDUIT = "Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT"

Use SDE Connection Name:

sde_connection = "Database Connections\\Connection to DWGIS3.sde\\"

ARG_CONDUIT = sde_connection + "ARG.CONDUIT"

Cleaned up names:

# Local variables:

file_gdb = "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\"

simple_conduits = file_gdb + "simple_conduits"

dissolved_conduits = file_gdb + "dissolved_conduits"

N0_Zero_Conduits = file_gdb + "N0_Zero_Conduits"

sde_connection = "Database Connections\\Connection to DWGIS3.sde\\"

ARG_CONDUIT = sde_connection + "ARG.CONDUIT"

Congratulations!

PyWin:

PyWin

Python for Windows extensions

by

Mark Hammond

http://sourceforge.net/projects/pywin32/

pywin32-218.win32-py2.7.exe

Process Steps:

Delete Features from simple_conduits

Append from SDE into simple_conduits,

just COND_NUM and WATER_TYPE

Delete NO_Zero_Conduits

Select from simple_conduits into N0_Zero_Conduits

Delete dissolved_conduits

Dissolve N0_Zero_Conduits into dissolved_conduits

Fix up Process Steps:

• # Process: Append

• arcpy.Append_management(ARG_CONDUIT, simple_conduits, "NO_TEST", "COND_NUM \"COND_NUM\" true true false 2 Short 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE \"WATER_TYPE\" true true false 5 Text 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length \"Shape_Length\" false true true 8 Double 0 0 ,First,#", "")

Becomes…

• # Process: Append

• arcpy.Append_management(

• "'Database Connections\\Connection to DWGIS3.sde\\

• ARG.DistributionSystem\\ARG.CONDUIT'", simple_conduits,

• "NO_TEST", "COND_NUM \"COND_NUM\" true true false 2 Short 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE \"WATER_TYPE\" true true false 5 Text 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length \"Shape_Length\" false true true 8 Double 0 0 ,First,#", "")

Fix up Process Steps:

• # Process: Delete

• arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass")

Becomes…

• # Process: Delete

• arcpy.Delete_management(

• "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\

• NO_Zero_Conduits",

• "FeatureClass")

Fix up Process Steps:

• # Process: Delete

• arcpy.Delete_management(dissolved_conduits, "FeatureClass")

Becomes…

• # Process: Delete

• arcpy.Delete_management(

• "R:\\ARG_Projects\\Conduit MapBook\\DissolvedConduits.gdb\\

• dissolved_conduits",

• "FeatureClass")

All Scrubbed Up: # Process:

arcpy.DeleteFeatures_management(simple_conduits)

arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass")

arcpy.Delete_management(dissolved_conduits, "FeatureClass")

arcpy.Append_management(ARG_CONDUIT, simple_conduits, "NO_TEST", "COND_NUM \"COND_NUM\" true true false 2 Short 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,COND_NUM,-1,-1;WATER_TYPE \"WATER_TYPE\" true true false 5 Text 0 0 ,First,#,Database Connections\\Connection to DWGIS3.sde\\ARG.DistributionSystem\\ARG.CONDUIT,WATER_TYPE,-1,-1;Shape_Length \"Shape_Length\" false true true 8 Double 0 0 ,First,#", "")

arcpy.Select_analysis(simple_conduits, N0_Zero_Conduits, "\"COND_NUM\" <> 0")

arcpy.Dissolve_management(N0_Zero_Conduits, dissolved_conduits, "COND_NUM;WATER_TYPE", "", "SINGLE_PART", "DISSOLVE_LINES")

Geek Speak - If:

if arcpy.Exists(something):

# do something with it

if not arcpy.Exists(something):

# it does not exist

Avoid Errors:

if arcpy.Exists(dissolved_conduits):

arcpy.Delete_management(dissolved_conduits, "FeatureClass")

Becomes…

arcpy.Delete_management(dissolved_conduits, "FeatureClass")

Delete Processes: if arcpy.Exists(N0_Zero_Conduits):

arcpy.Delete_management(N0_Zero_Conduits, "FeatureClass")

if arcpy.Exists(dissolved_conduits):

arcpy.Delete_management(dissolved_conduits, "FeatureClass")

if arcpy.Exists(simple_conduits):

arcpy.DeleteFeatures_management(simple_conduits)

else:

print "Missing feature class -> " + simple_conduits

Alternative:

>>> from arcpy import env

>>> env.overwriteOutput = True

>>> env.workspace = “C:/EsriPress/Python/Data/”

Either way, explain:

>>> # There feature classes probably already exist.

>>> # They are left over from the last time we ran this script.

>>> # Deleting the old versions makes room for the new ones.

Geek Speak – Try / Catch:

>>> try:

>>> # .. To do something that might blow up

>>> # and it “throws” an Exception

>>> except Exception ex:

>>> print ex

>>>

Catch Exception:

>>> try:

>>> arcpy.Append_management(CONDUIT

>> except Exception as ex:

>>> print ex

Test Exception:

>>> try:

>>> bogus = 1.0 / 0.0 # forces exception

>>> arcpy.Append_management(ARG_CONDUIT,

>>> except Exception as ex:

>>> print ex

Logging: import logging

logfile = "C:/temp/demo.log"

logging.basicConfig(

level=logging.INFO,

filename=logfile)

logger = logging.getLogger()

logger.info("A swallow beats its wings 43 times every second.")

logger.warning("Go away or I shall taunt you a second time.")

logger.error("Run away! Run away!")

print "Done."

C:\temp\demo.log: INFO:A swallow beats its wings 43 times every second.

WARNING:Go away or I shall taunt you a second time.

ERROR:Run away! Run away!

Logging: logging.info(" checking for " + HYDRANT_BRANCH )

if arcpy.Exists(HYDRANT_BRANCH ):

try:

arcpy.Append_management(APPEND_HYD_BR, hyd_branch, ...)

logging.info(" HYDRANT_BRANCH_shp appended to SDE")

except Exception as ex:

logging.error(" HYDRANT_BRANCH_shp append FAILED")

logging.Exeception(ex)

Recap – Do These 3 Things…

1. Organize variable names

2. Trap Errors

3. Write a log file

"First shalt thou take out the Holy Pin.

Then shalt thou count to three, no more, no less.

Three shall be the number thou shalt count, and the number of the counting shall be three.”

Pythonic:

• Explicit is better than implicit.

• Simple is better than complex.

• Readability counts.

The Zen of Python, by Tim Peters, includes:

Geek Speak - Blocks:

>>> if arcpy.Exists(simple_conduits):

>>> arcpy.DeleteFeatures_management(simple_conduits)

>>> # maybe do something else

>>> # some additional processing

>>> else:

>>> print "Missing feature class -> " + simple_conduits

Duck Typing:

If it looks like a duck, quacks like a duck, etc.

Works with strings,

integers,

floating point,

lists,

etc.

Strings:

Enclosed in either ‘single’ or “double” quotes.

Concatenate using + sign.

>>> topic = "GIS"

>>> location = "Rockies“

>>> event = topic + " in the " + location

>>> print event

GIS in the Rockies

Strings: More built-in functions:

>>> event = "GIS in the Rockies"

>>> print event.lower()

>>> print event.upper()

>>> print event.title()

>>> string_length = len(event)

>>> print string_length

>>> print event.find("Rockies",0,string_length)

gis in the rockies

GIS IN THE ROCKIES

Gis In The Rockies

18

11

Triple quotes can span multiple

lines: “””The Dead Collector: Bring out yer dead.

Large Man: Here's one.

The Dead Collector: That'll be ninepence.

The Dead Body: I'm not dead.

The Dead Collector: 'Ere, he says he's not dead.

Large Man: Yes he is.

The Dead Body: I'm not.

The Dead Collector: He isn't.

Large Man: Well, he will be soon, he's very ill.

The Dead Body: I'm getting better. “””

Lists: >>> python_sketch = ['Dead Parrot', 'Confuse-a-Cat', 'Silly Walks', 'Four Yorkshiremen','Fish Slap'];

>>> # notice duck typing

>>> print python_sketch [0]

>>> print python_sketch [1]

>>> python_sketch.append('Limberjack Song')

>>> print len(python_sketch)

Dead Parrot

Confuse-a-Cat

6

For loop:

>>> for funny_bit sketch in python_sketch:

>>> print funny_bit + ' hahaha‘

Dead Parrot hahaha

Confuse-a-Cat hahaha

Silly Walks hahaha

Four Yorkshiremen hahaha

Fish Slap hahaha

Limberjack Song hahaha

Command Line Args:

>>> userid = arcpy.GetParameterAsText(0)

>>> if not userid:

>>> print ‘Usage: python.exe my_script.py userid password ‘

>>> pword = arcpy.GetParameterAsText(1)

>>> if not pword:

>>> print ‘Usage: python.exe my_script.py userid password ‘

Command Line Args:

C:\> D:\python27\python.exe my_script.py zeke password

Python.exe is located on D:\python27

my_script.py is some python code

Userid is zeke

My Oracle password is at the end

Command Line Args:

>>> userid = arcpy.GetParameterAsText(0)

>>> pword = arcpy.GetParameterAsText(1)

Oracle connection:

connection_string = userid + '/' + pword + '@dwsde'

# looks like this -> zeke/password@dwsde

import cx_Oracle

connection = cx_Oracle.connect(connection_string)

cursor = connection.cursor()

cursor.execute('select distinct water_type from conduit')

for result in cursor:

print result

Oracle connection:

cursor.execute('select distinct diameter from conduit')

allresults = cursor.fetchall()

for result in allresults:

print result

cursor.close()

connection.close()

print "Done."

Nee

Here is a shrubbery.

Not Habit Forming:

But is considered a “Gateway” language,

which frequently leads to more nerdy behavior.

Good Book:

Good Book:

GIS Day November 22:

See denverwater.org for directions...

Becomes…