october 4, 2011 joe cross

Useful Python Techniques:A brief introduction to List Comprehensions,

Functional Programming, and Generators

October 4, 2011Joe Cross

Contents1. Looping and List Comprehensions2. Filter, Map, Reduce (and lambda!)3. Functional Programming Basics + Decorators4. Generators

1. Looping and List Comprehensions

for(int i = 0; i < 5; i++){

cout << witty_banter(i) << endl;

}

Loopingcompared to Java/C++

Looping

for i in range(0, 10, 1):    a /= 2.0 for i in xrange(10):    a /= 2.0

for(int i = 0; i < 10; i++){    a /= 2.0;}

Python

Java/C++

LoopingRange + Iter

Looping

aList = [0, 1, 2, 'hello', 2**-4] for i in range(len(aList)):    print aList[i] for item in aList:    print item

Loopingstill not as fun as…

Looping

To double the values in a list and assign to a new variable:

winning_lottery_numbers = [0, 4, 3, 2, 3, 1] fake_lottery_numbers = []for i in range(len(winning_lottery_numbers)):    fake_lottery_numbers.append(2 * winning_lottery_numbers[i])  fake_lottery_numbers = []for number in winning_lottery_numbers:    fake_lottery_numbers.append(2 * number)

Even though it’s an improvement over the tedium of c++ et. al, we can still do better.

List ComprehensionsWoohooo!

List Comprehensions

winning_lottery_numbers = [0, 4, 3, 2, 3, 1] fake_lottery_numbers = [2*n for n in winning_lottery_numbers]

List Comprehensions allow us to do all sorts of things:•Single-function single-line code•Apply a function to each item of an iterable•Filter using conditionals•Cleanly nest loops

Syntax:

[<expression> for <value> in <collection> if <condition>]

List ComprehensionsDon’t nest too many!

List Comprehensions

Multi-variable functions in a single line using zip:

vec1 = [3, 10, 2]vec2 = [-20, 5, 1] dot_mul = [u*v for u, v in zip(vec1, vec2)]dot_prod = sum(dot_mul)

Filtering:

readings = [-1.2, 0.5, 12, 1.8, -9.0, 5.3] good_readings = [r for r in readings if r > 0]

Bad:

orig = [15, 30, 78, 91, 25] finals = [min(s, 100) for s in [f+5 for f in orig]]

2. Filter, Map, Reduce

Life = map(evolution, irreducible complexity)assert(sum(Life) == 42)

Filter

Filter, Map, Reduce

def isPos(number, lim = 1E-16):    return number > lim >>> a = [-1,2,-3,4,-5,6,-7,8,-9,10]>>> filter(isPos, a)[2, 4, 6, 8, 10]

>>> filter(not isPos, a) Traceback (most recent call last):  File "<pyshell#7>", line 1    filter(not isZero, a)TypeError: 'bool' object is not callable

Syntax:

result = filter(aFunction, aSequence)

Filter + Lambda

Filter, Map, Reduce

def isPos(number, lim = 1E-16):    return number > lim

>>> filter(lambda n: not isPos(n), a)[-1, -3, -5, -7, -9]

[fnName] = lambda [args]: expression

Syntax:

Lambda vs. def

Filter, Map, Reduce

def add(x, y):    return x + y Ladd = lambda x, y: x+y def printWords():    print "Words" LprintWords = lambda: print "Words"

So… why use lambda?

Filter, Map, Reduce

When using verbose function declaration it is often the case that the function’sverbose declaration can be verbose, even for functions that don’t require such verbosity.

def ispos(n):    return n > 0b = filter(ispos, aList)

Also, there are some valid concerns about namespace clutter and the like.Verbose verbose verbose.

b = filter(lambda n: n > 0, aList)

Vs.

b = []for a in aList:    if a > 0:        b.append(a)

Verbose

Not Verbose

Map

Filter, Map, Reduce

Compare to list comprehension:

winning_lottery_numbers = [0, 4, 3, 2, 3, 1]

fake_lottery_numbers = [2*n for n in winning_lottery_numbers]

fake_lottery_numbers = map(lambda n: 2*n, winning_lottery_numbers)

1.2.

Syntax:

result = map(aFunction, aSequence)

Reduce

Filter, Map, Reduce

Syntax:

result = reduce(aFunction, aSequence, [initial])

lambda factorial n: reduce(operator.mul, xrange(1, n))

NOTE: results get accumulated on the left, and new values applied to the right.so reduce(add, [1,2,3,4]) is processed as (((1+2)+3)4)

3. Functional Programming + Decorators

This isn’t your dad’sProcedural (imperative) programming

A Simple Example

def add(x, y):    return x + y def sub(x, y):    return x - y def mul(x, y):    return x * y def div(x, y):    return x / y def mod(x, y):    return x % y

def op(fn, x, y):    return fn(x, y)

Functional Programming

Nested FunctionsSpeed vs. Obfuscation

Functional Programming

def randNoGap(min_, max_):    #random.random() -> [0,1)    v = random.random()    return (max_ - min_) * v - min_ def randWithGap(min_, max_):    s = random.random()    v = randNoGap(min_, max_)    if s < 0.5:        return v    else:        return -v   #Same conditional using Python’s   #Ternary operator    #return v if s < 0.5 else -v 

def rand(min_, max_, hasGap = False):    if hasGap:        return randWithGap(min_, max_)    else:        return randNoGap(min_, max_)

Nested FunctionsSpeed vs. Obfuscation (Continued)

Functional Programming

def randomExplosion(minv, maxv, n):    particles = []    for _ in xrange(n):        vx = rand(minv, maxv, True)        vy = rand(minv, maxv, True)        vz = rand(minv, maxv, True)        vx2 = rand(minv, maxv, True)        vy2 = rand(minv, maxv, True)        vz2 = rand(minv, maxv, True)        r = rand(0,255,False)        g = rand(0,255,False)        b = rand(0,255,False)        mainParticle = [vx,vy,vz,r,g,b]        secondParticle = [vx2,vy2,vz2,r,g,b]        particleGroup = (mainParticle, secondParticle)        particles.append(particleGroup)    return particles

NO

Nested FunctionsSpeed vs. Obfuscation (Continued)

Functional Programming

What we’d like to do:

velocities = [rndV() for _ in xrange(6)]

What it actually looks like:

velocities = [rand(minv,maxv,True) for i in xrange(6)]

With a functional wrapper, we re-map:

rndV -> make_rand_fnc(minv, maxv, True)

rndV() -> make_rand_fnc(minv, maxv, True)()

Nested FunctionsSpeed vs. Obfuscation (Continued)

Functional Programming

def mkRand(min_, max_, hasGap = False):    def wrapper():        return rand(min_, max_, hasGap)    return wrapper

rand(minv, maxv, True)rand(minv, maxv, True)rand(minv, maxv, True) def rand(min_, max_, hasGap = False):

def randomExplosion(minv, maxv, n):    rVel = mkRand(minv, maxv, True)    rCol = mkRnd(0,255,False)    for _ in xrange(n):        vx = rVel()        vy = rVel()        vz = rVel()        vx2 = rVel()        vy2 = rVel()        vz2 = rVel()        r = rCol()        g = rCol()        b = rCol()

Nested FunctionsSpeed vs. Obfuscation (Continued)

Functional Programming

def randomExplosion(minv, maxv, n):    particles = []    rndV = mkRand(minv, maxv, True)    rndC = mkRnd(0,255,False)    for _ in xrange(n):        velocities = [rndV() for i in xrange(6)]        r,g,b = [rndC() for i in xrange(3)]        mainParticle = velocities[:3] + [r,g,b]        secondParticle = velocities[3:] + [r,g,b]        particleGroup = (mainParticle, secondParticle)        particles.append(particleGroup)    return particles

DecoratorsQuickly apply common tasks to methods

Decorators

Common pre + post function call tasks, such as:•Caching•Timing•Counting function calls•Access rights

@decoratordef myFunc(arg1):    print “arg1: “, arg1 myFunc = decorator(myFunc)

@f1(arg)@f2def func(): pass def func(): passfunc = f1(arg)(f2(func))

DecoratorsQuickly apply common tasks to methods

Decorators

def decorator(f):    print "This line is run once during func = decorator(func)"    def wrapper(*args, **kwargs):        print "This line is executed just before the function is called"        #Call the function        ret = f(*args, **kwargs)        print "This line is executed just after the function is called"        #Return the function's return        return ret    return wrapper @decoratordef foo(bar):    print bar

On running, we get this output:>>> ================================ RESTART ================================>>> This line is run once during func = decorator(func)>>> foo(1)This line is executed just before the function is called1This line is executed just after the function is called

DecoratorsQuickly apply common tasks to methods

Decorators

Decorators using classesclass decorator(object):    def __init__(self, f):        print "This line is run once during func = decorator(func)"        self.f = f     def __call__(self, *args, **kwargs):        print "This line is executed just before the function is called"

        #Call the function        ret = self.f(*args)

        print "This line is executed just after the function is called"

        #Return the function's return        return ret

DecoratorsQuickly apply common tasks to methods

Decorators

(Rough) Timing

import time class TIMED(object):    def __init__(self, f):        self.f = f

    def __call__(self, *args):        start = time.clock()        ret = self.f(*args)        stop = time.clock()        print "{0}: {1} ms.".format(self.f.func_name, 1000*(stop-start))        return ret

DecoratorsQuickly apply common tasks to methods

Decorators

@TIMEDdef euler(f, t0, y0, h):   """ Euler's Method """   yn = y0 + h*f(t0,y0)   return yn @TIMEDdef RK2(f, t0, y0, h):   """ Heun's Method """   y_hat = y0 + h*f(t0,y0)   yn = y0 + h/2.0*(f(t0,y0)+f(t0+h, y_hat))   return yn @TIMEDdef RK4(f, t0, y0, h):   """ Standard RK4 """   k1 = f(t0, y0)   k2 = f(t0+h/2.0,y0 + h*k1/2.0)   k3 = f(t0+h/2.0,y0 + h*k2/2.0)   k4 = f(t0+h/2.0,y0 + h*k3)   yn = y0 + 1.0/6.0*h*(k1 + 2.0*k2 + 2.0*k3 + k4)   return yn

fns = [euler, RK2, RK3, RK4, jRK4]t0 = scipy.linspace(-1,1)y0 = scipy.ones(50)h = 0.025args = (f, t0, y0, h) for fn in fns:    print fn(*args)    print

DecoratorsQuickly apply common tasks to methods

Decorators

>>> euler: 0.0181114469778 ms.[ ... ] RK2: 0.041656328049 ms.[  ... ] RK3: 0.0606733473757 ms.[  ... ] RK4: 0.0745587900587 ms.[ ... ] jRKN: 0.00150928724815 ms.jRK4: 1.57358288492 ms.[ ... ]

4. Generators

Memory-conscious patternsare kind of a big deal in scientific computing

Binary Tree from Array(simplified interface)

class T(object):    def __init__(self, values = None, index = 0):        self.left = None        self.right = None        self.v = None        if values is not None:            self.loadValues(values, index)     def loadValues(self, values, index):        self.v = values[index]        n = len(values)        if index * 2 + 1 < n:            self.left = T(values, index * 2 + 1)        if index * 2 + 2 < n:            self.right = T(values, index * 2 + 2)

Generators

Guessing Game

def makeT(val, delta, levels, level = 0):    if level < levels:        t = T()        t.v = val        t.left = makeT(val-delta, delta/2, levels, level+1)        t.right = makeT(val+delta, delta/2, levels, level+1)        return t 

Generators

Clean Code

def inorder(t):    if t:        for v in inorder(t.left):            yield v        yield t.v        for v in inorder(t.right):            yield v

Generators

Using our Generator

for v in inorder(a_tree):    print v a = []for v in inorder(a_tree):    a.append(v) b = [v for v in inorder(a_tree)]

Generators

Questions?

“Yes, the slide with the code. What did that one do?”