Python FoundationsPython Foundations

by Chris Gahanby Chris Gahan

(( In Stereo where Available ))(( In Stereo where Available ))

TRICK #1TRICK #1

Don’t Repeat YourselfDon’t Repeat Yourself If everything is defined once, then If everything is defined once, then

radically changing the behaviour of your radically changing the behaviour of your program is trivial.program is trivial.

TRICK #2TRICK #2

Get your program running on real Get your program running on real data as soon as possibledata as soon as possible

TRICK #3TRICK #3

Test it!Test it! Even if you don’t have fancy test cases, make Even if you don’t have fancy test cases, make

some little bit of code at the end of the some little bit of code at the end of the program to test its functionality as you’re program to test its functionality as you’re developing it.developing it.

I like to break my programs into a bunch of I like to break my programs into a bunch of modules with tests in the ‘if __name__ == modules with tests in the ‘if __name__ == “__main__”:’ block“__main__”:’ block

when you execute the file, the tests runwhen you execute the file, the tests run when you import it as a module, the tests are ignoredwhen you import it as a module, the tests are ignored

TRICK #4TRICK #4

Use as much of Python’s libraries and Use as much of Python’s libraries and built-in functionality as you canbuilt-in functionality as you can Python’s built-in stuff is a solid foundationPython’s built-in stuff is a solid foundation There are solutions to many many There are solutions to many many

problems in the problems in the standard librarystandard library Use lists and dicts for as much as you canUse lists and dicts for as much as you can

writing custom classes when you can use a list writing custom classes when you can use a list or dict will overcomplicate your codeor dict will overcomplicate your code

TRICK #5TRICK #5

The Zen of PythonThe Zen of Python Execute “import this” in the interpreterExecute “import this” in the interpreter

Things you should knowThings you should know

Exceptions are awesomeExceptions are awesome But don’t over-use themBut don’t over-use them

Decorators are coolDecorators are cool But you rarely need themBut you rarely need them

List comprehensions and Generators are List comprehensions and Generators are wickedwicked Use them as much as possible!Use them as much as possible! Rethinking your problems as a series of generators Rethinking your problems as a series of generators

or list transformations is an excellent habit to get or list transformations is an excellent habit to get intointo

More things you should know More things you should know (about performance)(about performance)

Appending strings is slowAppending strings is slow Use lists and ‘’.join(list) them!Use lists and ‘’.join(list) them!

Searching lists is slowSearching lists is slow Use sets or dicts!Use sets or dicts!

Comparing lists is slowComparing lists is slow Use sets!Use sets!

List comprehensions are fast!List comprehensions are fast! Use them as much as possibleUse them as much as possible

If your program still isn’t fast enough, use Psyco If your program still isn’t fast enough, use Psyco or Pyrex!or Pyrex! Psyco optimizes code automatically (faster)Psyco optimizes code automatically (faster) Pyrex generates C code which can be compiled (fastest)Pyrex generates C code which can be compiled (fastest)

Everything is an objectEverything is an object

Lists are objectsLists are objects Functions are objectsFunctions are objects Classes are objectsClasses are objects Types are objects (IntType, ClassType, Types are objects (IntType, ClassType,

TypeType, etc.)TypeType, etc.) Dicts are objectsDicts are objects Objects are objectsObjects are objects Internally, all Objects are secretly Dicts!Internally, all Objects are secretly Dicts!

ListsLists This is your bread and butter in Python. Lists are This is your bread and butter in Python. Lists are

everything:everything: list = [1,2,3,4,Ninja(),”rumplestiltskin”]list = [1,2,3,4,Ninja(),”rumplestiltskin”]

Handy methods:Handy methods: list.append(5)list.append(5)

Stick 5 on the end of the listStick 5 on the end of the list list.extend([5,6,7])list.extend([5,6,7])

Append the contents of [5,6,7] to the listAppend the contents of [5,6,7] to the list list.pop(list.pop(positionposition))

Yank off an element from the list and hand it to you. If you don’t Yank off an element from the list and hand it to you. If you don’t specify the position, it pops off the last element.specify the position, it pops off the last element.

list.reverse()list.reverse() Reverse the listReverse the list

list.sort()list.sort() Sort the listSort the list

list.index(item)list.index(item) Return the position of Return the position of item item in the listin the list

SlicingSlicing Lets you grab chunks from a list or a string.Lets you grab chunks from a list or a string.

list[5:9]list[5:9] Grabs elements 5,6,7,8Grabs elements 5,6,7,8

list[-3:-1]list[-3:-1] Negative indexes mean “from the end”, -1 being first from Negative indexes mean “from the end”, -1 being first from

the end, -2 being second from the end, etc.. So this would the end, -2 being second from the end, etc.. So this would grab two elements: the 3grab two elements: the 3rdrd and 2 and 2ndnd from the end. from the end.

list[:4]list[:4] Grabs elements 0,1,2,3Grabs elements 0,1,2,3

list[6:]list[6:] Start at element 6 and grab all the rest of the elements Start at element 6 and grab all the rest of the elements

until the enduntil the end list[:]list[:]

Grab all elements (make a copy of the entire thing)Grab all elements (make a copy of the entire thing)

Note: The off-by-one thing (where it ignores the last element in the slice) is there for a good reason – it makes slice math super easy! (It’s the same reason that range(a,b) ignores the last element ‘b’)

Slicing StringsSlicing Strings

Since strings are also technically lists Since strings are also technically lists of characters, they can also be slicedof characters, they can also be sliced

To copy a string, take a slice that’s To copy a string, take a slice that’s the entire string:the entire string: newString = oldString[:]newString = oldString[:]

TriviaTrivia

Did you know that slices are actually Did you know that slices are actually OBJECTS? It’s TRUE! OBJECTS? It’s TRUE! You can make an object that returns its You can make an object that returns its

own custom slice objects which return own custom slice objects which return anything you wantanything you want

SQLObject does this when you slice a SQLObject does this when you slice a database query result – it uses “LIMIT” and database query result – it uses “LIMIT” and “OFFSET” to get the chunk of the slice, “OFFSET” to get the chunk of the slice, resulting in resulting in mad speed!mad speed!

Stupid Tuple TricksStupid Tuple Tricks

Tuples are like lists, but staticTuples are like lists, but static once you define a tuple, you can’t modify itonce you define a tuple, you can’t modify it

Whenever you use a comma in Python, a Whenever you use a comma in Python, a tuple is secretly created. For example:tuple is secretly created. For example: a = 1, 2, 3a = 1, 2, 3 open(“file.txt”, “r”)open(“file.txt”, “r”) person1, person2 = “Muhammed”, “Shareef”person1, person2 = “Muhammed”, “Shareef” a, b = b, aa, b = b, a

Internally, function parameters are tuples.

A clever way to swap two variables

Assigning two values at once

Kinds of StringsKinds of Strings

Different kinds of strings:Different kinds of strings:

Raw strings ignore escape codes (\n, \x, etc)Raw strings ignore escape codes (\n, \x, etc) Great for regexes – you don’t have to go: Great for regexes – you don’t have to go: \\t\\t, , \\w\\w, , \\s\\s, ,

etc..etc..

String operationsString operations

s.replace(‘victim’, ‘replacement’)s.replace(‘victim’, ‘replacement’) Replace all occurrences of victimReplace all occurrences of victim

s.strip([chars])s.strip([chars]) Remove leading/trailing whitespace and newlines Remove leading/trailing whitespace and newlines

(or, remove leading/trailing [chars])(or, remove leading/trailing [chars]) s.split([delimiter])s.split([delimiter])

Break a string up into pieces and return them as a Break a string up into pieces and return them as a list of strings. The delimiter is whitespace by list of strings. The delimiter is whitespace by default.default.

You want more? Hit TAB in ipython!You want more? Hit TAB in ipython!

Formatting Text for OutputFormatting Text for Output

Want to format text for output?Want to format text for output? If you have a bunch of elements you If you have a bunch of elements you

want to print nicely, use want to print nicely, use “, “.join(list)“, “.join(list) If you want to print strings, instead of:If you want to print strings, instead of:

““hi “ + name + “, how are you?”hi “ + name + “, how are you?” ……you can do:you can do:

““hi %s, how are you?” % namehi %s, how are you?” % name OrOr

““hi %(name)s, how are you?” % locals()hi %(name)s, how are you?” % locals()

String InterpolationString Interpolation

String interpolation lets you substitute String interpolation lets you substitute placeholders (%s, %d, etc.) in a string (just placeholders (%s, %d, etc.) in a string (just like printf in C)like printf in C)

Why string interpolation?Why string interpolation?

Because when you have lots of variables in the output, Because when you have lots of variables in the output, it becomes annoying to keep track of where your it becomes annoying to keep track of where your quotes end. (Also, typing “++“ requires hitting shift a quotes end. (Also, typing “++“ requires hitting shift a lot – it’s easy to mess it up). lot – it’s easy to mess it up). ““name: “+name+”, age: “+age+”, city: “+city+”, province: name: “+name+”, age: “+age+”, city: “+city+”, province:

“+province+”, IP: “+ip+”, favorite colour: “+favorite_colour“+province+”, IP: “+ip+”, favorite colour: “+favorite_colour ……is harder to read than:is harder to read than:

““name: %s, age: %s, city: %s, province: %s, IP: %s, favorite name: %s, age: %s, city: %s, province: %s, IP: %s, favorite colour: %s” % (name, age, city, province, ip, favorite_colour)colour: %s” % (name, age, city, province, ip, favorite_colour)

……which is harder to read than:which is harder to read than: ““name: %(name)s, age: %(age)s, city: %(city)s, province: %name: %(name)s, age: %(age)s, city: %(city)s, province: %

(province)s, IP: %(ip)s, favorite colour: %(favorite_colour)s” % (province)s, IP: %(ip)s, favorite colour: %(favorite_colour)s” % locals()locals()

Of course, Ruby has the prettiest string interpolation:Of course, Ruby has the prettiest string interpolation: ““name: #{name}, age: #{age}, city: #{city}, etc..” name: #{name}, age: #{age}, city: #{city}, etc..”

FilesFiles

Files are great in pythonFiles are great in python f = open(“filename.txt”)f = open(“filename.txt”)

Making lists prettyMaking lists pretty

Even though Python usually cares about Even though Python usually cares about indentation, lists don’t… so get creative!indentation, lists don’t… so get creative!

Note: You can have an extra comma on the last item to make cutting and pasting and shifting lines around easier!

Making dicts prettyMaking dicts pretty

Dicts let you use that extra comma too…

Making function calls prettyMaking function calls pretty

Sometimes you need a huge number of Sometimes you need a huge number of function parameters. There’s a couple function parameters. There’s a couple ways to make this readable:ways to make this readable:

There’s that extra comma again!

FunctionsFunctions Standard functionStandard function

You can set the default value for a parameter (in this case, None).You can set the default value for a parameter (in this case, None). These are called These are called keyword argumentskeyword arguments

““if” statementsif” statements

You can do some neat tricks in “if” You can do some neat tricks in “if” statements.statements. Compare identity with the “is” and “is not” Compare identity with the “is” and “is not”

operators:operators: if theSocket is not None:if theSocket is not None: if hand.contents() is None:if hand.contents() is None:

Checking if stuff is empty:Checking if stuff is empty: if []:if []: => false=> false if “”:if “”: => false=> false if None: if None: => false=> false if 0:if 0: => false=> false

““if” statementsif” statements

There’s also the handy “elif” construct:There’s also the handy “elif” construct:(That’s right, Python has no switch statement!)

FunctionsFunctions

Inline functions are kinda gross and Inline functions are kinda gross and very limited (they can only execute very limited (they can only execute one statement, and it’s returned by one statement, and it’s returned by default), but they’re sometimes default), but they’re sometimes handy.handy. It’s usually better to just “def” a It’s usually better to just “def” a

function.function.

(List comprehensions are better for this, by the way…)

Capturing Function Capturing Function ParametersParameters

You can capture all the parameters passed You can capture all the parameters passed to a function in a list by naming one to a function in a list by naming one parameter *args:parameter *args:

Capturing Function Capturing Function ParametersParameters

You can also get a dictionary of all You can also get a dictionary of all the the keyword argumentskeyword arguments

Using the “Apply” methodUsing the “Apply” method

Apply lets you call a function and pass the Apply lets you call a function and pass the parameters in as a list. For example, using the parameters in as a list. For example, using the “cmp” (compare) function (which returns -1 “cmp” (compare) function (which returns -1 for less than, 0 for equal, 1 for greater than):for less than, 0 for equal, 1 for greater than):

But you don’t actually But you don’t actually needneed apply…apply…

You can actually do the same thing You can actually do the same thing by putting the parameters in a list by putting the parameters in a list and “dereferencing” it with *list:and “dereferencing” it with *list:

And it works for dicts too!And it works for dicts too!

Dictionaries can be dereferenced by Dictionaries can be dereferenced by doing **dict:doing **dict:

RegexesRegexes

Regexes can be executed on a string directly:Regexes can be executed on a string directly: match = re.match(‘expression’, ‘string’)match = re.match(‘expression’, ‘string’)

OR, they can be compiled first for extra speed:OR, they can be compiled first for extra speed: compiled_regex = re.compile(‘expression’)compiled_regex = re.compile(‘expression’) match = compiled_regex.match(‘string’)match = compiled_regex.match(‘string’)

When you do a .match() or a .search():When you do a .match() or a .search(): No match returns No match returns NoneNone A match returns a re match object, which contains:A match returns a re match object, which contains:

groups()groups() group([group number])group([group number]) groupdict() (if you used named groups: “(?groupdict() (if you used named groups: “(?

<name>expression)”)<name>expression)”)

DecoratorsDecorators

Decorators let you add wrapper code Decorators let you add wrapper code around methods and classesaround methods and classes

The Old Way

The New Way

Require Integer DecoratorRequire Integer Decorator

Pre/Postconditions Pre/Postconditions DecoratorDecorator

AutomaticAutomaticThreadThreadLockingLocking

DecoratorDecorator

Static Methods / Class Static Methods / Class MethodsMethods

Oh yeah, I forgot to mention…Oh yeah, I forgot to mention… Static methods are methods that you Static methods are methods that you

can call on an instance or the class itselfcan call on an instance or the class itself They don’t take “self” as a parameter, so They don’t take “self” as a parameter, so

they can’t operate on the object. They’re they can’t operate on the object. They’re just helper methods.just helper methods.

Class methods are called on classes only Class methods are called on classes only and receive the class as the first and receive the class as the first argument (instead of “self”)argument (instead of “self”)

Useful when you have a big class hierarchy Useful when you have a big class hierarchy and you want the parent class to do things and you want the parent class to do things to the child classes.to the child classes.

Docstrings, PyDoc, iPythonDocstrings, PyDoc, iPython

Docstrings rule.Docstrings rule. It’s why everything in ipython has help It’s why everything in ipython has help

for itfor it Whoever invented them was a genius, Whoever invented them was a genius,

because they’re elegant and powerful:because they’re elegant and powerful:

(It’s common to use triple-quotes because they let you write multi-linedocstrings, and they kinda look nicer, but “regular strings” work too.)

List ComprehensionsList Comprehensions List comprehensions make it simpler to do a really List comprehensions make it simpler to do a really

common task: computing something for every common task: computing something for every element in a list and putting the result in another element in a list and putting the result in another list.list. Check out that SAVINGS!Check out that SAVINGS!

List ComprehensionsList Comprehensions

They can also work as filters.They can also work as filters. Here’s how you filter a list of numbers Here’s how you filter a list of numbers

and return only the even numbers:and return only the even numbers:

GeneratorsGenerators

Generators are special objects that Generators are special objects that spit out values one at a time as spit out values one at a time as requested by the caller.requested by the caller. They can result in huge memory savings They can result in huge memory savings

when processing large batches of datawhen processing large batches of data They’re also great for implementing They’re also great for implementing

things like pipes, or procedurally things like pipes, or procedurally generated sequencesgenerated sequences

GeneratorsGenerators To make your function a generator, To make your function a generator,

just use “just use “yield <value>yield <value>” instead of ” instead of ““return <value>return <value>””

Calling a generator returns a Calling a generator returns a generator objectgenerator object

Generator objects have a Generator objects have a .next().next() method that returns the next value in method that returns the next value in the sequencethe sequence

Generators maintain their state Generators maintain their state between calls to between calls to .next().next()

GeneratorsGenerators

Example:Example:

GeneratorsGenerators

Another example:Another example:

Generator ExpressionsGenerator Expressions

Identical to list comprehensions, Identical to list comprehensions, except that the result is a generator except that the result is a generator instead of a list.instead of a list. for square in ( num**2 for num in for square in ( num**2 for num in

numbers ):numbers ): print “you got yourself a square! => ”, print “you got yourself a square! => ”,

squaresquare

More tips…More tips…

Making programs flexibleMaking programs flexible

It makes it much easier to write code when It makes it much easier to write code when you can change it easilyyou can change it easily

I constantly modify my programs as I’m I constantly modify my programs as I’m going, and as such, I’ve developed some going, and as such, I’ve developed some good tricks to make that easier.good tricks to make that easier.

Writing your program as if it’s an API that Writing your program as if it’s an API that you can call from the interpreter is an easy you can call from the interpreter is an easy way of testing itway of testing it see altavista.pysee altavista.py