WTF is Twisted?(or: Owl Amongst The Ponies)

About HawkOwl• hawkowl@atleastfornow.net

• @hawkieowl on Twitter

• hawkowl on GitHub, Keybase & ThoughtStreams

• PGP Key Fingerprint:2100 5027 897E C8AE 1D94A905 2308 B479 D392 4A11

• Twisted Core Committer and Release Manager (13.2 & 14.0)

• Is not actually an owl

Twisted in One Slide

• Twisted is a framework for writing network applications in Python

• Best known for “Deferreds”, Twisted’s async abstraction

• Contains protocol implementations for HTTP, SMTP, IMAP, DNS, etc.

Why would you want to use Twisted?

• Higher performance in some common situations

• Stable, mature codebase

• Protocol implementations for lots of things

• High test coverage

• Great community consisting of excellent individuals who do excellent work

What makes Twisted especially different from other projects?

• Twisted has a reputation for being the “black sheep” in the Python community, mostly because we do things “weirdly” and “hard”

• But, surprise! It turns out the Twisted developers were right all along (see: asyncio)

• Different set of principals, from both the longevity of the project and their specific problems

The Tenets of Twisted(the time-tested, battle-hardened principals of the

Twisted project, as interpreted by HawkOwl)

Subclassing is bad.Composition is the new inheritance!

Multiple inheritance is really bad.

Handy hint: If you’ve written the word “super”, it’s already too late.

Time-based releases are king.

For large, evolving frameworks, it’s better to gradually change than have a massive app migration effort.

Early APIs > No APISYou won’t know how your API works in the real world until

other people use it. You can always deprecate it later.

All code needs tests. No exceptions.

It doesn’t matter if you’re fixing an urgent release regression - an eye on coverage saves you a lot of time later.

All code needs reviewing. No exceptions.

Code review is invaluable, even if it does descend into bikeshedding if they can’t find anything wrong.

Only bad APIs can’t be unit tested.

Can’t unit test your API? Too bad, time to refactor it.

Build slaves are amazing.

Test your code on everything. Virtual machines are cheap, compatibility problems down the line are not.

–David Reid, Glyph, et al

“Call a function with some arguments and inspect some return values.”

How Twisted Works(a short, mostly-correct overview)

The Event Loop(Spin me right round, baby, right round)

The Reactor• Twisted, being an event-driven framework,

provides an event loop, or “reactor”.

• This is an infinite loop, wrapped around a blocking call that waits until some I/O is ready

• Once the I/O is ready, this fires off an event to run the code that was waiting for it

• Runs other tasks while I/O blocked tasks are waiting

Using the Reactor• To utilise the reactor, you use Deferreds.

• Deferreds are an “IOU” for an actual result sometime in the future

• A Deferred contains a callback chain, a series of functions run in sequence after it gets a result

• Deferreds get their results from events occurring - such as I/O being received or some task being finished

Benefits

• For I/O bound applications, allows you to efficiently use CPU resources, as the reactor runs other things in the meantime

• Integration with common event loops - for example, GTK or QT, to use your Twisted code into your GTK or QT applications

The Callback Chain(Hey, I just met you, and this is crazy,

but here’s a function, so call it maybe?)

The Callback Chain

• The easiest way to understand the callback chain is to visualise it as a production line.

• A production line is made up of discrete blocks - each doing a separate thing.

• Let’s imagine a such a production line…

Lets say that we are building a toy car, and we need to do the following:

1. Create toy car body from materials

2. Paint racing stripes on car body

3. Add wheels to the car body

4. Put the completed car in a box

Here is how we would do that in psuedo-code Deferreds: !

! d = Deferred()! !d.addCallback(buildCarBody)! d.addCallback(paintStripes)! d.addCallback(addWheels)! d.addCallback(placeInBox)! # Start the callback chain! d.callback(rawMaterials)

Deferreds

• Each function in the chain is called with the result of the previous one as its first argument

• Functions in the callback chain can return either a result or a Deferred that fires with a result sometime in the future

Lists of Deferreds

• If you have several Deferred operations to do at the same time, and want your callback to fire only when they are all done, use DeferredList.!

• Pass it a list of Deferreds, and it will return a Deferred that fires with a list containing the results of each when they have all finished.

Error Handling

• Error handling is done by errbacks.

• Errbacks are like callbacks, but are run when an exception is raised.

• An errback will catch all errors that occur on the chain before it.

Deferred Benefits

• Forces you to break up your logic into sections, broken up over external interface accesses

• Encourages reuse, as these sections quite commonly do one thing to an input and return it

• Can be cancelled (eg. if a client disconnected and you don’t care about the result anymore)

Deferred Benefits

• Discourages global mutable state by having different execution threads share the same global state

• No global mutable state means you can scale easier

• If your global state is databases, just spin up more Twisted instances to use all your CPUs

Extending the Callback Chain(Because sometimes things like to do other things)

Deferreds returning Deferreds

• Deferreds can contain their own callback chain - meaning that one Deferred returned may have several callbacks being called inside it

• This is invisible to the “top level” of callbacks

• You can extend your callback chain at any time

What can Twisted do?(Enough of the boring stuff!)

Web• Klein, a Flask-like API in Twisted

• Saratoga, a HTTP API framework

• Autobahn, a Websockets implementation

• Hendrix, a Django deployment framework

• Static file serving: twistd -n web --port tcp:8080 --path /tmp

Klein (Flask in Twisted)

from klein import Klein!app = Klein()!!

@app.route('/')!def home(request):! return 'Hello, world!'!!

app.run("localhost", 8080)

Saratoga (HTTP API){! "metadata": {! "name": “APIExample", "versions": [1]! },! "endpoints": [{!! ! "endpoint": “example", "getProcessors": [! {"versions": [1]}! ]}!]}!

from saratoga.api import SaratogaAPI!import json!!class APIExample(object):! class v1(object):! def example_GET(self, request, params):! return {"hello": "world"}!!api = SaratogaAPI(APIExample, json.load(open("apidef.json")))!api.run()

Email

• Complete SMTP and ESMTP server and clients

• POP3 and IMAP4 servers and clients

• Support for TLS-secured email out of the boxFor more info, see: http://twistedmatrix.com/documents/current/mail/index.html

DNS• A caching, recursive DNS server out of the boxtwistd -n dns -c —port 10053 --recursive

• DNS client APIs

• DNS server toolkitFor more info, see: http://twistedmatrix.com/documents/current/names/howto/custom-server.html

Chat• IRC Server twistd -n words --irc-port tcp:8400 --group roomname --auth memory:usr:pass

• IRC client (for making IRC bots)

• XMPP clientFor more info, see: http://twistedmatrix.com/documents/current/words/examples/

Want to know more?

• Twisted Docshttp://twistedmatrix.com/documents/current/

• Iffy’s Twisted FTWhttp://iffycan.com/twistedftw/

• HawkOwl’s Technicalshttp://technicals.atleastfornow.net/

Questions?(sane answers not guaranteed)