generators: the final frontier
DESCRIPTION
PyCon'2014 tutorial presentation.TRANSCRIPT
![Page 1: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/1.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators: The Final Frontier
David Beazley (@dabeaz)http://www.dabeaz.com
Presented at PyCon'2014, Montreal
1
![Page 2: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/2.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
2
• Generator Tricks for Systems Programmers (2008)
http://www.dabeaz.com/generators/
• A flying leap into generator awesomeness
![Page 3: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/3.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
3
• A Curious Course on Coroutines and Concurrency (2009)
http://www.dabeaz.com/coroutines/
• Wait, wait? There's more than iteration?
![Page 4: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/4.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Today's Installment
4
• Everything else you ever wanted to know about generators, but were afraid to try
• Part 3 of a trilogy
![Page 5: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/5.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Requirements
5
• You need Python 3.4 or newer
• No third party extensions
• Code samples and notes
http://www.dabeaz.com/finalgenerator/
• Follow along if you dare!
![Page 6: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/6.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Disclaimer
6
• This is an advanced tutorial
• Assumes general awareness of
• Core Python language features
• Iterators/generators
• Decorators
• Common programming patterns
• I learned a LOT preparing this
![Page 7: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/7.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Will I Be Lost?
7
• Although this is the third part of a series, it's mostly a stand-alone tutorial
• If you've seen prior tutorials, that's great
• If not, don't sweat it
• Be aware that we're focused on a specific use of generators (you just won't get complete picture)
![Page 8: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/8.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Focus
8
• Material in this tutorial is probably not immediately applicable to your day job
• More thought provoking and mind expanding
• from __future__ import future
practicalutility
bleedingedge
![Page 9: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/9.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part I
9
Preliminaries - Generators and Coroutines(rock)
![Page 10: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/10.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators 101• yield statement defines a generator function
10
def countdown(n): while n > 0: yield n n -= 1
• You typically use it to feed iterationfor x in countdown(10): print('T-minus', x)
• A simple, yet elegant idea
![Page 11: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/11.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Under the Covers
11
• Generator object runs in response to next()>>> c = countdown(3)>>> c<generator object countdown at 0x10064f900>>>> next(c)3>>> next(c)2>>> next(c)1>>> next(c)Traceback (most recent call last): File "<stdin>", line 1, in ?StopIteration>>>
• StopIteration raised when function returns
![Page 12: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/12.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
12
• Generators as "iterators" misses the big picture
• There is so much more to yield
![Page 13: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/13.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators as Pipelines• Stacked generators result in processing pipelines
• Similar to shell pipes in Unix
13
generatorinput sequence
for x in s:generator generator
def process(sequence): for s in sequence: ... do something ... yield item
• Incredibly useful (see prior tutorial)
![Page 14: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/14.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines 101• yield can receive a value instead
14
def receiver(): while True: item = yield print('Got', item)
• It defines a generator that you send things torecv = receiver()next(recv) # Advance to first yieldrecv.send('Hello')recv.send('World')
![Page 15: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/15.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines and Dataflow
15
• Coroutines enable dataflow style processing
source coroutine
coroutine
send() send()
• Publish/subscribe, event simulation, etc.
coroutine
coroutinesend()
send()
coroutinesend()
![Page 16: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/16.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Fundamentals
• The yield statement defines a generator function
16
def generator(): ... ... yield ... ...
• The mere presence of yield anywhere is enough
• Calling the function creates a generator instance>>> g = generator()>>> g<generator object generator at 0x10064f120>>>>
![Page 17: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/17.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advancing a Generator• next(gen) - Advances to the next yield
17
def generator(): ... ... yield item ...
• Returns the yielded item (if any)
• It's the only allowed operation on a newly created generator
• Note: Same as gen.__next__()
![Page 18: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/18.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Sending to a Generator
• gen.send(item) - Send an item to a generator
18
def generator(): ... item = yield ... ... yield value
• Wakes at last yield, returns sent value
• Runs to the next yield and emits the value
g = generator()next(g) # Advance to yield
value = g.send(item)
![Page 19: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/19.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Closing a Generator• gen.close() - Terminate a generator
19
def generator(): ... try: yield except GeneratorExit: # Shutting down ...
• Raises GeneratorExit at the yield
• Only allowed action is to return
• If uncaught, generator silently terminates
g = generator()next(g) # Advance to yield
g.close() # Terminate
![Page 20: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/20.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Raising Exceptions• gen.throw(typ [, val [,tb]]) - Throw exception
20
def generator(): ... try: yield except RuntimeError as e: ... ... yield val
• Raises exception at yield
• Returns the next yielded value (if any)
g = generator()next(g) # Advance to yield
val = g.throw(RuntimeError, 'Broken')
![Page 21: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/21.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Return Values• StopIteration raised on generator exit
21
def generator(): ... yield ... return result
• Return value (if any) passed with exception
• Note: Python 3 only behavior (in Python 2, generators can't return values)
g = generator()try: next(g)except StopIteration as e: result = e.value
![Page 22: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/22.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Delegation• yield from gen - Delegate to a subgenerator
22
def generator(): ... yield value ... return result
def func(): result = yield from generator()
• Allows generators to call other generators
• Operations take place at the current yield
• Return value (if any) is returned
![Page 23: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/23.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Delegation Example• Chain iterables together
23
def chain(x, y): yield from x yield from y
• Example:>>> a = [1, 2, 3]>>> b = [4, 5, 6]>>> for x in chain(a, b):... print(x,end=' ')... 1 2 3 4 5 6
>>> c = [7,8,9]>>> for x in chain(a, chain(b, c)):... print(x, end=' ')...1 2 3 4 5 6 7 8 9>>>
![Page 24: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/24.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Mini-Reference
• Generator instance operations
24
gen = generator()
next(gen) # Advance to next yieldgen.send(item) # Send an itemgen.close() # Terminategen.throw(exc, val, tb) # Raise exceptionresult = yield from gen # Delegate
• Using these, you can do a lot of neat stuff
• Generator definitiondef generator(): ... yield ... return result
![Page 25: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/25.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 2
25
and now for something completely different
![Page 26: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/26.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Common Motif• Consider the following
26
f = open()...f.close()
lock.acquire()...lock.release()
db.start_transaction()...db.commit()
start = time.time()...end = time.time()
• It's so common, you'll see it everywhere!
![Page 27: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/27.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Managers• The 'with' statement
27
with open(filename) as f: statement statement ...
with lock: statement statement ...
• Allows control over entry/exit of a code block
• Typical use: everything on the previous slide
![Page 28: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/28.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• It's easy to make your own (@contextmanager)
28
import timefrom contextlib import contextmanager
@contextmanagerdef timethis(label): start = time.time() try: yield finally: end = time.time() print('%s: %0.3f' % (label, end-start)
• This times a block of statements
![Page 29: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/29.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
29
• Usage
with timethis('counting'): n = 1000000 while n > 0: n -= 1
• Outputcounting: 0.023
![Page 30: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/30.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management• Another example: temporary directories
30
import tempfile, shutilfrom contextlib import contextmanager
@contextmanagerdef tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir)
• Examplewith tempdir() as dirname: ...
![Page 31: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/31.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Whoa, Whoa, Stop!• Another example: temporary directories
31
import tempfile, shutilfrom contextlib import contextmanager
@contextmanagerdef tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir)
• Examplewith tempdir() as dirname: ...
What is this?• Not iteration• Not dataflow• Not concurrency• ????
![Page 32: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/32.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Under the covers
32
with obj: statements statements statements ... statements
• If an object implements these methods it can monitor entry/exit to the code block
obj.__enter__()
obj.__exit__()
![Page 33: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/33.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager• Implementation template
33
class Manager(object): def __enter__(self): return value def __exit__(self, exc_type, val, tb): if exc_type is None: return else: # Handle an exception (if you want) return True if handled else False
• Use:with Manager() as value: statements statements
![Page 34: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/34.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager Example• Automatically deleted temp directories
34
import tempfileimport shutil
class tempdir(object): def __enter__(self): self.dirname = tempfile.mkdtemp() return self.dirname
def __exit__(self, exc, val, tb): shutil.rmtree(self.dirname)
• Use:with tempdir() as dirname: ...
![Page 35: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/35.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Alternate Formulation• @contextmanager is just a reformulation
35
import tempfile, shutilfrom contextlib import contextmanager
@contextmanagerdef tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname)
• It's the same code, glued together differently
![Page 36: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/36.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction• How does it work?
36
@contextmanagerdef tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname)
• Think of "yield" as scissors
• Cuts the function in half
![Page 37: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/37.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction• Each half maps to context manager methods
37
@contextmanagerdef tempdir(): dirname = tempfile.mkdtemp() try: yield dirname
statements statements statements ...
finally: shutil.rmtree(dirname)
__enter__
__exit__
user statements ('with' block)
• yield is the magic that makes it possible
![Page 38: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/38.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction• There is a wrapper class (Context Manager)
38
class GeneratorCM(object): def __init__(self, gen): self.gen = gen
def __enter__(self): ...
def __exit__(self, exc, val, tb): ...
• And a decoratordef contextmanager(func): def run(*args, **kwargs): return GeneratorCM(func(*args, **kwargs)) return run
![Page 39: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/39.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction• enter - Run the generator to the yield
39
class GeneratorCM(object): def __init__(self, gen): self.gen = gen
def __enter__(self): return next(self.gen)
def __exit__(self, exc, val, tb): ...
• It runs a single "iteration" step
• Returns the yielded value (if any)
![Page 40: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/40.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction• exit - Resumes the generator
40
class GeneratorCM(object): ... def __exit__(self, etype, val, tb): try: if etype is None: next(self.gen) else: self.gen.throw(etype, val, tb) raise RuntimeError("Generator didn't stop") except StopIteration: return True except: if sys.exc_info()[1] is not val: raise
• Either resumes it normally or raises exception
![Page 41: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/41.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Full Disclosure
• Actual implementation is more complicated
• There are some nasty corner cases
• Exceptions with no associated value
• StopIteration raised inside a with-block
• Exceptions raised in context manager
• Read source and see PEP-343
41
![Page 42: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/42.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Discussion
• Why start with this example?
• A completely different use of yield
• Being used to reformulate control-flow
• It simplifies programming for others (easy definition of context managers)
• Maybe there's more... (of course there is)
42
![Page 43: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/43.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
43
Call me, maybe
![Page 44: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/44.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
44
Call me, maybe
![Page 45: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/45.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Async Processing• Consider the following execution model
45
def func(args): ... ... ... return result
run_async(func, args)
main thread
• Examples: Run in separate process or thread, time delay, in response to event, etc.
![Page 46: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/46.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example: Thread Pool
46
from concurrent.futures import ThreadPoolExecutor
def func(x, y): 'Some function. Nothing too interesting' import time time.sleep(5) return x + y pool = ThreadPoolExecutor(max_workers=8)fut = pool.submit(func, 2, 3)r = fut.result()print('Got:', r)
• Runs the function in a separate thread
• Waits for a result
![Page 47: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/47.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
47
>>> fut = pool.submit(func, 2, 3)>>> fut<Future at 0x1011e6cf8 state=running>>>>
• Future - A result to be computed later
• You can wait for the result to return>>> fut.result()5>>>
• However, this blocks the caller
![Page 48: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/48.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
48
def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler)
def result_handler(fut): result = fut.result() print('Got:', result)
• Alternatively, you can register a callback
• Triggered upon completion
![Page 49: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/49.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exceptions
49
>>> fut = pool.submit(func, 2, 'Hello')>>> fut.result()Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 395, in result return self.__get_result() File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result raise self._exception File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run result = self.fn(*self.args, **self.kwargs) File "future2.py", line 6, in func return x + yTypeError: unsupported operand type(s) for +: 'int' and 'str'>>>
![Page 50: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/50.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures w/Errors
50
def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler)
def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e))
• Error handling with callbacks
• Exception propagates out of fut.result() method
![Page 51: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/51.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
51
def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler)
def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e))
• Consider the structure of code using futures
• Meditate on it... focus on the code.
• This seems sort of familiar
![Page 52: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/52.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Callback Hell?
52
• No, no, no.... keep focusing.
![Page 53: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/53.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
53
def entry(): fut = pool.submit(func, 2, 3) fut.add_done_callback(exit)
def exit(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e))
• What if the function names are changed?
• Wait! This is almost a context manager (yes)
![Page 54: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/54.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Futures
54
@inlined_futuredef do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result)
run_inline_future(do_func)
• Thought: Maybe you could do that yield trick
• The extra callback function is eliminated
• Now, just one "simple" function
• Inspired by @contextmanager
![Page 55: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/55.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Déjà Vu
55
![Page 56: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/56.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Déjà Vu
56
• This twisted idea has been used before...
![Page 57: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/57.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Preview
57
t = Task(gen)
• There are two separate parts
• Part 1: Wrapping generators with a "task"
• Part 2: Implementing some runtime coderun_inline_future(gen)
• Forewarning: It will bend your mind a bit
![Page 58: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/58.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
58
• Will continue to use threads for examples
• Mainly because they're easy to work with
• And I don't want to get sucked into an event loop
• Don't dwell on it too much
• Key thing: There is some background processing
![Page 59: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/59.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
59
def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result)
• Problem: Stepping through a generator
• Involves gluing callbacks and yields together
def do_func(x, y): yield pool.submit(func, x, y)
result = print('Got:', result)
add_done_callback()
cut
enter
exit
![Page 60: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/60.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
60
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): result = fut.result()! self.step(result)
![Page 61: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/61.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
61
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): result = fut.result()! self.step(result)
Task class wraps around and represents a running generator.
![Page 62: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/62.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
62
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): result = fut.result()! self.step(result)
Advance the generator to the next
yield, sending in a value (if any)
![Page 63: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/63.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
63
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): result = fut.result()! self.step(result)
Attach a callback to the produced Future
![Page 64: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/64.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
64
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): result = fut.result()! self.step(result)
Collect result and send back into the
generator
![Page 65: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/65.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
65
pool = ThreadPoolExecutor(max_workers=8)
def func(x, y): time.sleep(1) return x + y
def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result)
t = Task(do_func(2, 3))t.step()
• Try it:
• Output:Got: 5
• Yes, it works
Note: must initiate first step of the task
to get it to run
![Page 66: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/66.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
66
pool = ThreadPoolExecutor(max_workers=8)
def func(x, y): time.sleep(1) return x + y
def do_many(n): while n > 0: result = yield pool.submit(func, n, n) print('Got:', result) n -= 1
t = Task(do_many(10))t.step()
• More advanced: multiple yields/looping
• Yes, this works too.
![Page 67: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/67.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
67
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc)
![Page 68: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/68.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
68
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc)
send() or throw() depending on
success
![Page 69: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/69.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
69
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass
def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc)
Catch exceptions and pass to next
step as appropriate
![Page 70: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/70.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Error Example
70
def do_func(x, y): try: result = yield pool.submit(func, x, y) print('Got:', result) except Exception as e: print('Failed:', repr(e))
t = Task(do_func(2, 'Hello'))t.step()
• Try it:
• Output:Failed: TypeError("unsupported operand type(s) for +: 'int' and 'str'",)
• Yep, that works too.
![Page 71: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/71.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
71
• This whole thing is rather bizarre
• Execution of the inlined future takes place all on its own (concurrently with other code)
• The normal rules don't apply
![Page 72: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/72.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Consider
72
def recursive(n): yield pool.submit(time.sleep, 0.001) print('Tick:', n) Task(recursive(n+1)).step()
Task(recursive(0)).step()
• Infinite recursion?
• Output:Tick: 0Tick: 1Tick: 2...Tick: 1662773Tick: 1662774...
![Page 73: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/73.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 4
73
yield from yield from yield from yield from future(maybe)
source: @UrsulaWJ
![Page 74: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/74.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
74
• Focus on the future
• Not the past
• Not now
• Yes, the future.
• No, really, the future.
(but not the singularity)
![Page 75: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/75.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
75
class Task: def __init__(self, gen): self._gen = gen
def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass...
generator must only produce Futures
![Page 76: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/76.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
76
def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• It's trying to delay the execution of a user-supplied inlined future until later.
![Page 77: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/77.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
77
def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• NoTraceback (most recent call last):... AttributeError: 'generator' object has no attribute 'add_done_callback'
![Page 78: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/78.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
78
def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen
Task(after(10, do_func(2, 3))).step()
• Can you make library functions?
• This is busted
• gen is a generator, not a Future
![Page 79: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/79.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
79
def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f
Task(after(10, do_func(2, 3))).step()
• What about this?
• Idea: Just iterate the generator manually
• Make it produce the required Futures
![Page 80: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/80.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
80
def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f
Task(after(10, do_func(2, 3))).step()
• What about this?
• No luck. The result gets lost somewhereGot: None
• Hmmm.
![Page 81: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/81.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
81
def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass
Task(after(10, do_func(2, 3))).step()
• Obvious solution (duh!)
• Hey, it works!Got: 5
![Page 82: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/82.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
82
def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass
Task(after(10, do_func(2, 3))).step()
• Obvious solution (duh!)
• Hey, it works!Got: 5
manual running of generator with results (ugh!)
![Page 83: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/83.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (4th Attempt)
83
def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen
Task(after(10, do_func(2, 3))).step()
• A better solution: yield from
• 'yield from' - Runs the generator for you
Got: 5
• And it works! (yay!)
• Awesome
![Page 84: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/84.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
PEP 380• yield from gen - Delegate to a subgenerator
84
def generator(): ... yield value ... return result
def func(): result = yield from generator()
• Transfer control to other generators
• Operations take place at the current yield
• Far more powerful than you might think
![Page 85: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/85.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Conundrum
85
def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen
• "yield" and "yield from"?
• Two different yields in the same function
• Nobody will find that confusing (NOT!)
![Page 86: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/86.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
86
def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen
Task(after(10, do_func(2, 3))).step()
• Maybe this will work?
• Just use 'yield from'- always!
![Page 87: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/87.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
87
def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen
Task(after(10, do_func(2, 3))).step()
• Maybe this will work?
• Just use 'yield from'- always!
• No. 'yield' and 'yield from' not interchangeable:Traceback (most recent call last):...TypeError: 'Future' object is not iterable>>>
![Page 88: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/88.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
??????
88
(Can it be made to work?)
![Page 89: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/89.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Iterable Futures
89
def patch_future(cls): def __iter__(self): ! if not self.done(): yield self return self.result() cls.__iter__ = __iter__
from concurrent.futures import Futurepatch_future(Future)
• A simple ingenious patch
• It makes all Future instances iterable
• They simply produce themselves and the result
• It magically makes 'yield from' work!
![Page 90: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/90.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
All Roads Lead to Future
90
yield self
• Future is the only thing that actually yields
• Everything else delegates using 'yield from'
• Future terminates the chain
generator
generator
generator
Future
yield from
yield from
yield from
run()
next(gen)
![Page 91: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/91.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Decorator
91
import inspectdef inlined_future(func): assert inspect.isgeneratorfunction(func) return func
• Generators yielding futures is its own world
• Probably a good idea to have some demarcation
• Does nothing much at all, but serves as syntax@inlined_futuredef after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen
• Alerts others about what you're doing
![Page 92: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/92.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Wrangling
92
t = Task(gen)t.step()
• The "Task" object is just weird
• No way to obtain a result
• No way to join with it
• Or do much of anything useful at all
Task
runs
????
t.step()
![Page 93: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/93.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
93
class Task(Future): def __init__(self, gen): !super().__init__()! self._gen = gen
def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value)
• This tiny tweak makes it much more interesting
![Page 94: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/94.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
94
class Task(Future): def __init__(self, gen): !super().__init__()! self._gen = gen
def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value)
• This tiny tweak makes it much more interesting
A Task is a Future
Set its result upon completion
![Page 95: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/95.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
95
@inlined_futuredef do_func(x, y): result = yield pool.submit(func, x, y) return result
t = Task(do_func(2,3))t.step()...print("Got:", t.result())
• Obtaining the result of task
• So, you create a task that runs a generator producing Futures
• The task is also a Future
• Right. Got it.
![Page 96: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/96.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
96
@inlined_futuredef do_func(x, y): result = yield pool.submit(func, x, y) return result
t = Task(do_func(2,3))t.step()...print("Got:", t.result())
class Task(Future): ... def step(self, value=None, exc=None): try: ... except StopIteration as exc: self.set_result(exc.value) ...
![Page 97: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/97.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Runners
97
def start_inline_future(fut): t = Task(fut) t.step() return t
def run_inline_future(fut): t = start_inline_future(fut) return t.result()
• You can make utility functions to hide details
result = run_inline_future(do_func(2,3))print('Got:', result)
• Example: Run an inline future to completion
• Example: Run inline futures in parallelt1 = start_inline_future(do_func(2, 3))t2 = start_inline_future(do_func(4, 5))result1 = t1.result()result2 = t2.result()
![Page 98: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/98.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Step Back Slowly
98
• Built a generator-based task system for threads
Tasks@inline_future
run_inline_future()
Threadspools
concurrent.future
Futures
submit
result
• Execution of the future hidden in background
• Note: that was on purpose (for now)
![Page 99: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/99.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
99
• Ideas are the foundation asyncio coroutines
Tasks@coroutine
run_until_complete()Event Loop
asyncio
Futures
result
• In fact, it's almost exactly the same
• Naturally, there are some details with event loop
![Page 100: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/100.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Simple Example
100
import asyncio
def func(x, y): return x + y
@asyncio.coroutinedef do_func(x, y): yield from asyncio.sleep(1) return func(x, y)
loop = asyncio.get_event_loop()result = loop.run_until_complete(do_func(2,3))print("Got:", result)
• asyncio "hello world"
![Page 101: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/101.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
101
import asyncio
@asyncio.coroutinedef echo_client(reader, writer): while True: line = yield from reader.readline() if not line: break resp = b'Got:' + line writer.write(resp) writer.close()
loop = asyncio.get_event_loop()loop.run_until_complete( asyncio.start_server(echo_client, host='', port=25000))loop.run_forever()
• asyncio - Echo Server
![Page 102: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/102.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Be on the Lookout!
102
![Page 103: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/103.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 103(source: globalpost.com)
![Page 104: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/104.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 5
104
"Gil"
![Page 105: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/105.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Python Threads
105
• Threads, what are they good for?
• Answer: Nothing, that's what!
• Damn you GIL!!
![Page 106: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/106.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actually...
106
• Threads are great at doing nothing!time.sleep(2) # Do nothing for awhile
data = sock.recv(nbytes) # Wait around for datadata = f.read(nbytes)
• In fact, great for I/O!
• Mostly just sitting around
![Page 107: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/107.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
107
• Threads are weak for computation
• Global interpreter lock only allows 1 CPU
• Multiple CPU-bound threads fight each other
• Could be better
http://www.dabeaz.com/GIL
![Page 108: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/108.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Solution
108
• Naturally, we must reinvent the one thing that threads are good at
• Namely, waiting around.
• Event-loops, async, coroutines, green threads.
• Think about it: These are focused on I/O
(yes, I know there are other potential issues with threads, but work with me here)
![Page 109: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/109.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
109
• Event-loops have their own issues
• Don't bug me, I'm blocking right now(source: chicagotribune.com)
![Page 110: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/110.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Standard Solution
110
• Delegate the work out to a process pool
python
python
python
python
python
main program
CPU
CPU
CPU
CPU
• multiprocessing, concurrent.futures, etc.
![Page 111: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/111.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
111
• Didn't we just do this with inlined futures?def fib(n): return 1 if n <= 2 else (fib(n-1) + fib(n-2))
@inlined_futuredef compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result
pool = ProcessPoolExecutor(4)result = run_inline_future(compute_fibs(35))
• It runs without crashing (let's ship it!)
![Page 112: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/112.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
112
• Can you launch tasks in parallel?t1 = start_inline_future(compute_fibs(34))t2 = start_inline_future(compute_fibs(34))result1 = t1.result()result2 = t2.result()
• Sequential executionrun_inline_future(compute_fibs(34))run_inline_future(compute_fibs(34))
• Recall (from earlier)def start_inline_future(fut): t = Task(fut) t.step() return t
![Page 113: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/113.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
113
• Can you launch tasks in parallel?t1 = start_inline_future(compute_fibs(34))t2 = start_inline_future(compute_fibs(34))result1 = t1.result()result2 = t2.result()
• Sequential executionrun_inline_future(compute_fibs(34))run_inline_future(compute_fibs(34))
• Recall (from earlier)def start_inline_future(fut): t = Task(fut) t.step() return t
9.56s
![Page 114: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/114.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
114
• Can you launch tasks in parallel?t1 = start_inline_future(compute_fibs(34))t2 = start_inline_future(compute_fibs(34))result1 = t1.result()result2 = t2.result()
• Sequential executionrun_inline_future(compute_fibs(34))run_inline_future(compute_fibs(34)) 9.56s
• Recall (from earlier)def start_inline_future(fut): t = Task(fut) t.step() return t
4.78s
Inlined tasks running outside confines of
the GIL?
![Page 115: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/115.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Execution Model
115
• The way in which it works is a little odd@inlined_futuredef compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result
addthis
• Output: (2 Tasks)<_MainThread(MainThread, started 140735086636224)><_MainThread(MainThread, started 140735086636224)><Thread(Thread-1, started daemon 4320137216)><Thread(Thread-1, started daemon 4320137216)><Thread(Thread-1, started daemon 4320137216)>...
????
![Page 116: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/116.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Process Pools
116
• Process pools involve a hidden result thread
main thread python
python
CPU
CPU
submit
result_thread
• result thread reads returned values
• Sets the result on the associated Future
• Triggers the callback function (if any)
results
![Page 117: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/117.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Issue
117
• Our inlined future switches execution threads
@inlined_futuredef compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result
main thread
result thread
• Switch occurs at the first yield
• All future execution occurs in result thread
• That could be a little weird (especially if it blocked)
![Page 118: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/118.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Important Lesson
118
• If you're going to play with control flow, you must absolutely understand possible implications under the covers (i.e., switching threads across the yield statement).
![Page 119: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/119.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Insight
119
• The yield is not implementation
@inlined_futuredef compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result
• You can implement different execution models
• You don't have to follow a formulaic rule
![Page 120: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/120.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Thread Execution
120
• Variant: Run generator entirely in a single threaddef run_inline_thread(gen): value = None exc = None while True: try: if exc: fut = gen.throw(exc) else: fut = gen.send(value) try: value = fut.result() exc = None except Exception as e: exc = e except StopIteration as exc: return exc.value
• It just steps through... no callback function
![Page 121: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/121.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
121
• Try it again with a thread pool (because why not?)@inlined_futuredef compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result
tpool = ThreadPoolExecutor(8)t1 = tpool.submit(run_inline_thread(compute_fibs(34)))t2 = tpool.submit(run_inline_thread(compute_fibs(34)))result1 = t1.result()result2 = t2.result()
![Page 122: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/122.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
122
• Output: (2 Threads)<Thread(Thread-1, started 4319916032)><Thread(Thread-2, started 4326428672)><Thread(Thread-1, started 4319916032)><Thread(Thread-2, started 4326428672)><Thread(Thread-1, started 4319916032)><Thread(Thread-2, started 4326428672)>...
(works perfectly)
4.60s
(a bit faster)
• Processes, threads, and futures in perfect harmony
• Uh... let's move along. Faster. Must go faster.
![Page 123: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/123.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Big Idea
123
• You can mold and adapt generator execution
• That yield statement: magic!
![Page 124: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/124.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 6
124
Fake it until you make it
![Page 125: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/125.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actors
125
• There is a striking similarity between coroutines and actors (i.e., the "actor" model)
• Features of Actors
• Receive messages
• Send messages to other actors
• Create new actors
• No shared state (messages only)
• Can coroutines serve as actors?
![Page 126: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/126.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
126
• A very simple example@actordef printer(): while True: msg = yield print('printer:', msg)
printer()n = 10while n > 0: send('printer', n) n -=1
idea: use generators to define a kind of "named" actor task
![Page 127: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/127.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Attempt 1
127
_registry = { }
def send(name, msg): _registry[name].send(msg)
def actor(func): def wrapper(*args, **kwargs): gen = func(*args, **kwargs) next(gen) _registry[func.__name__] = gen return wrapper
• Make a central coroutine registry and a decorator
• Let's see if it works...
![Page 128: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/128.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
128
• A very simple example@actordef printer(): while True: msg = yield print('printer:', msg)
printer()n = 10while n > 0: send('printer', n) n -=1
• It seems to work (maybe)printer: 10printer: 9printer: 8...printer: 1
![Page 129: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/129.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
129
• Recursive ping-pong (inspired by Stackless)@actordef ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1)
@actordef pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1)
ping()pong()send('ping', 0)
ping
pong
![Page 130: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/130.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
130
ping 0pong 1Traceback (most recent call last): File "actor.py", line 36, in <module> send('ping', 0) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 24, in ping send('pong', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 31, in pong send('ping', n + 1) File "actor.py", line 8, in send _registry[name].send(msg)ValueError: generator already executing
• Alas, it does not work
![Page 131: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/131.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Problems
131
• Important differences between actors/coroutines
• Concurrent execution
• Asynchronous message delivery
• Although coroutines have a "send()", it's a normal method call
• Synchronous
• Involves the call-stack
• Does not allow recursion/reentrancy
![Page 132: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/132.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 1
132
class Actor(threading.Thread): def __init__(self, gen): super().__init__() self.daemon = True self.gen = gen self.mailbox = Queue() self.start()
def send(self, msg): self.mailbox.put(msg)
def run(self): while True: msg = self.mailbox.get() self.gen.send(msg)
• Wrap the generator with a thread
• Err...... no.
![Page 133: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/133.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 2
133
_registry = { }_msg_queue = deque()
def send(name, msg): _msg_queue.append((name, msg))
def run(): while _msg_queue: name, msg = _msg_queue.popleft() _registry[name].send(msg)
• Write a tiny message scheduler
• send() simply drops messages on a queue
• run() executes as long as there are messages
![Page 134: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/134.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
134
• Recursive ping-pong (reprise)@actordef ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1)
@actordef pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1)
ping()pong()send('ping', 0)run()
ping
pong
![Page 135: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/135.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
135
ping 0pong 1ping 2pong 3ping 4ping 5ping 6pong 7...... forever
• It works!
• That's kind of amazing
![Page 136: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/136.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Comments
136
• It's still kind of a fake actor
• Lacking in true concurrency
• Easily blocked
• Maybe it's good enough?
• I don't know
• Key idea: you can bend space-time with yield
![Page 137: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/137.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 7
137
A Terrifying Visitor
![Page 138: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/138.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Let's Write a Compiler
138
• Well, an extremely simple one anyways...
• Evaluating mathematical expressions
2 + 3 * 4 - 5
• Why?
• Because eval() is for the weak, that's why
![Page 139: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/139.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
139
• Lexing : Make tokens
2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
• Parsing : Make a parse tree[NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM]
PLUS
NUM(2)
TIMES
NUM(3)
NUM(4)
MINUS
NUM(5)
![Page 140: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/140.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
140
• Evaluation : Walk the parse tree
PLUS
NUM(2)
TIMES
NUM(3)
NUM(4)
MINUS
NUM(5) 9
• It's almost too simple
![Page 141: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/141.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
141
import refrom collections import namedtupletokens = [ r'(?P<NUM>\d+)', r'(?P<PLUS>\+)', r'(?P<MINUS>-)', r'(?P<TIMES>\*)', r'(?P<DIVIDE>/)', r'(?P<WS>\s+)', ]
master_re = re.compile('|'.join(tokens))Token = namedtuple('Token', ['type','value'])
def tokenize(text): scan = master_re.scanner(text) return (Token(m.lastgroup, m.group()) for m in iter(scan.match, None) if m.lastgroup != 'WS')
![Page 142: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/142.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
142
text = '2 + 3 * 4 - 5'for tok in tokenize(text): print(tok)
Token(type='NUM', value='2')Token(type='PLUS', value='+')Token(type='NUM', value='3')Token(type='TIMES', value='*')Token(type='NUM', value='4')Token(type='MINUS', value='-')Token(type='NUM', value='5')
• Example:
![Page 143: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/143.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Parsing
143
• Must match the token stream against a grammarexpr ::= term { +|- term }*term ::= factor { *|/ factor}*factor ::= NUM
• An expression is just a bunch of terms2 + 3 * 4 - 5
term + term - term
• A term is just one or more factorsterm term
factor factor * factor(2) (3) (4)
![Page 144: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/144.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
144
expr ::= term { +|- term }*term ::= factor { *|/ factor}*factor ::= NUM
def expr():! term()! while accept('PLUS','MINUS'): term() !print('Matched expr')
def term():! factor() while accept('TIMES','DIVIDE'): factor() print('Matched term')
def factor(): if accept('NUM'): print('Matched factor') else: raise SyntaxError()
Encode the grammaras a collection of
functions
Each function steps through the rule
![Page 145: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/145.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
145
def parse(toks): lookahead, current = next(toks, None), None def accept(*toktypes): nonlocal lookahead, current if lookahead and lookahead.type in toktypes: current, lookahead = lookahead, next(toks, None) return True
def expr(): term() while accept('PLUS','MINUS'): term() print('Matched expr') ... expr()
![Page 146: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/146.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
146
class Node: _fields = [] def __init__(self, *args): for name, value in zip(self._fields, args): setattr(self, name, value)
class BinOp(Node): _fields = ['op', 'left', 'right']
class Number(Node): _fields = ['value']
• Need some tree nodes for different things
• Example:n1 = Number(3)n2 = Number(4)n3 = BinOp('*', n1, n2)
![Page 147: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/147.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
147
def parse(toks): ... def expr(): left = term() while accept('PLUS','MINUS'): left = BinOp(current.value, left) left.right = term() return left
def term(): left = factor() while accept('TIMES','DIVIDE'): left = BinOp(current.value, left) left.right = factor() return left
def factor(): if accept('NUM'): return Number(int(current.value)) else: raise SyntaxError() return expr()
Building nodes and hooking them together
![Page 148: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/148.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Our Little Parser
148
text = '2 + 3*4 - 5'toks = tokenize(text)tree = parse(toks)
• Story so far...
BinOp('-', BinOp('+', Number(2), BinOp('*', Number(3), Number(4) ) ), Number(5))
![Page 149: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/149.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
149
class NodeVisitor: def visit(self, node): return getattr(self, 'visit_' + type(node).__name__)(node)
• The "Visitor" pattern
• Example:class MyVisitor(NodeVisitor): def visit_Number(self, node): print(node.value) def visit_BinOp(self, node): self.visit(node.left) self.visit(node.right) print(node.op)
MyVisitor().visit(tree)
234*+5-
output
![Page 150: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/150.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
150
class Evaluator(NodeVisitor): def visit_Number(self, node):! return node.value
def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
9
![Page 151: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/151.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Digression
151
• Last 12 slides a whole graduate CS course
• Plus at least one additional Python tutorial
• Don't worry about it
• Left as an exercise...
![Page 152: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/152.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Death Spiral
152
# Make '0+1+2+3+4+...+999'text = '+'.join(str(x) for x in range(1000))toks = tokenize(text)tree = parse(toks)val = Evaluate().visit(tree)
• And it almost works...
Traceback (most recent call last): File "compiler.py", line 100, in <module> val = Evaluator().visit(tree) File "compiler.py", line 63, in visit return getattr(self, 'visit_' + type(node).__name__)(node) File "compiler.py", line 80, in visit_BinOp leftval = self.visit(node.left) ...RuntimeError: maximum recursion depth exceeded while calling a Python object
![Page 153: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/153.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
153
class Evaluator(NodeVisitor): def visit_Number(self, node):! return node.value
def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
!%*@*^#^#Recursion
(damn you to hell)
![Page 154: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/154.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
154
+
0 + 1 + 2 + 3 + 4 ... + 999
0
+
+
+
+
2
3
998999
1
...
A deeplynested tree structure
Blows up!
![Page 155: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/155.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155
• The visitor pattern is bad idea
• Better: Functional language with pattern matching and tail-call optimization
I Told You So
![Page 156: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/156.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
156
How do you NOT do something?
![Page 157: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/157.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
157
How do you NOT do something?
(yield?)
![Page 158: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/158.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
158
class Evaluator(NodeVisitor): def visit_Number(self, node):! return node.value
def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval
print(Evaluator().visit(tree))
• An Expression Evaluator
Nope. Not doing that recursion.
![Page 159: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/159.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
159
class NodeVisitor: def genvisit(self, node): result = getattr(self, 'visit_' + type(node).__name__)(node) if isinstance(result, types.GeneratorType): result = yield from result return result
• Step 1: Wrap "visiting" with a generator
• Thinking: No matter what the visit_() method produces, the result will be a generator
• If already a generator, then just delegate to it
![Page 160: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/160.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
160
>>> v = Evaluator()>>> n = Number(2)>>> gen = v.genvisit(n)>>> gen<generator object genvisit at 0x10070ab88>>>> gen.send(None)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 2>>>
• Example: A method that simply returns a value
• Result: Carried as value in StopIteration
![Page 161: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/161.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
161
>>> v = Evaluator()>>> n = BinOp('*', Number(3), Number(4))>>> gen = v.genvisit(n)>>> gen<generator object genvisit at 0x10070ab88>>>> gen.send(None)<__main__.Number object at 0x1058525c0>>>> gen.send(_.value)<__main__.Number object at 0x105852630>>>> gen.send(_.value)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 12>>>
• A method that yields nodes (iteration)
Again, note the return mechanism
![Page 162: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/162.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
162
>>> v = Evaluator()>>> n = BinOp('*', Number(3), Number(4))>>> gen = v.genvisit(n)>>> gen<generator object genvisit at 0x10070ab88>>>> gen.send(None)<__main__.Number object at 0x1058525c0>>>> gen.send(_.value)<__main__.Number object at 0x105852630>>>> gen.send(_.value)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 12>>>
• A method that yields nodes
def visit_Number(self, node): return node.value
Manually carrying out this method in the example
![Page 163: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/163.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
163
class NodeVisitor: def visit(self, node): stack = [ self.genvisit(node) ]! result = None while stack: try: node = stack[-1].send(result) stack.append(self.genvisit(node)) result = None except StopIteration as exc:! stack.pop() result = exc.value! return result
• Step 2: Run depth-first traversal with a stack
• Basically, a stack of running generators
![Page 164: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/164.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Transcendence
164
# Make '0+1+2+3+4+...+999'text = '+'.join(str(x) for x in range(1000))toks = tokenize(text)tree = parse(toks)val = Evaluate().visit(tree)print(val)
• Does it work?
• Yep499500
• Yow!
![Page 165: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/165.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
165
class Evaluator(NodeVisitor): def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': result = leftval + rightval ... return result
• Each yield creates a new stack entry
• Returned values (via StopIteration) get propagated as results
generator
generator
generator
stack
yield
yield return
return
![Page 166: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/166.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
166
>>> v = Evaluator()>>> n = BinOp('*', Number(3), Number(4))>>> stack = [ v.genvisit(n) ]>>> stack[-1].send(None)<__main__.Number object at 0x1058525c0>>>> stack.append(v.genvisit(_))>>> stack[-1].send(None)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 3>>> stack.pop()>>> stack[-1].send(3)<__main__.Number object at 0x105852630>>>> stack.append(v.genvisit(_))>>> stack[-1].send(None)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 4>>> stack.pop()>>> stack[-1].send(4)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 12>>>
Nodes are visited and generators pushed onto a stack
![Page 167: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/167.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
167
>>> v = Evaluator()>>> n = BinOp('*', Number(3), Number(4))>>> stack = [ v.genvisit(n) ]>>> stack[-1].send(None)<__main__.Number object at 0x1058525c0>>>> stack.append(v.genvisit(_))>>> stack[-1].send(None)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 3>>> stack.pop()>>> stack[-1].send(3)<__main__.Number object at 0x105852630>>>> stack.append(v.genvisit(_))>>> stack[-1].send(None)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 4>>> stack.pop()>>> stack[-1].send(4)Traceback (most recent call last): File "<stdin>", line 1, in <module>StopIteration: 12>>>
Results propagate via StopIteration
12 (Final Result)
![Page 168: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/168.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 168
Final Words
![Page 169: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/169.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Historical Perspective
169
• Generators seem to have started as a simple way to implement iteration (Python 2.3)
• Took an interesting turn with support for coroutines (Python 2.5)
• Taken to a whole new level with delegation support in PEP-380 (Python 3.3).
![Page 170: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/170.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Control Flow Bending
170
• yield statement allows you to bend control-flow to adapt it to certain kinds of problems
• Wrappers (context managers)
• Futures/concurrency
• Messaging
• Recursion
• Frankly, it blows my mind.
![Page 171: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/171.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
171
• Inclusion of asyncio in standard library may be a game changer
• To my knowledge, it's the only standard library module that uses coroutines/generator delegation in a significant manner
• To really understand how it works, need to have your head wrapped around generators
• Read the source for deep insight
![Page 172: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/172.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Is it Proper?
172
• Are coroutines/generators a good idea or not?
• Answer: I still don't know
• Issue: Coroutines seem like they're "all in"
• Fraught with potential mind-bending issues
• Example: Will there be two standard libraries?
![Page 173: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/173.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Two Libraries?
173
PythonStandard Library
Standardcoroutine library
(asyncio and friends)
?????
• If two different worlds, do they interact?
• If so, by what rules?
![Page 174: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/174.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Personal Use
174
• My own code is dreadfully boring
• Generators for iteration: Yes.
• Everything else: Threads, recursion, etc. (sorry)
• Nevertheless: There may be something to all of this advanced coroutine/generator business
![Page 175: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/175.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Bit More Information
175
![Page 176: Generators: The Final Frontier](https://reader036.vdocument.in/reader036/viewer/2022081412/53fdcd598d7f72a81c8b4b2e/html5/thumbnails/176.jpg)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thanks!
176
• I hope you got some new ideas
• Please feel free to contact me
http://www.dabeaz.com
• Also, I teach Python classes (shameless plug)
@dabeaz (Twitter)
• Special Thanks:
Brian Curtin, Ken Izzo, George Kappel, Christian Long, Michael Prentiss, Vladimir Urazov, Guido van Rossum