asynchronous python with twisted
TRANSCRIPT
Asynchronous
Python with Twisted
Harnessing the power of one
Who Am I?
Developer/Evangelist at LaunchKey
Organizer of Las Vegas Python User Group
Founder/Co-Organizer of Las Vegas PHP UG
Co-Organizer of Las Vegas Developer UG
National Junior Basketball Coach
What is Asynchronous Programming?• Non-Linear
• Event Driven
• Functional
• Depends on non-blocking code
Why Asynchronous?
One thread for all connections with the following benefits:
Shared application level objects and data reduces memory footprint per connection
Limited threads allows for better CPU utilization of each core thus fewer cores overall.
Both of the above enable large applications to run on smaller virtual machines in the cloud.
What’s the Catch?
Functional non-blocking code development has a learning curve.
Developing applications with objects that persist beyond a single request can be more difficult than standard WSGI applications.
Requires asynchronous libraries which can be scarce in Python.
Twisted
Established Asynchronous Python Framework
1.0 Release was in March 2004
Utilizes asynchronous I/O
Server runs in Python 2.6+ and 3.2+
Deferred = Asynchronous
Analogous to promises/futures
Returned by asynchronous and concurrent calls
Use callbacks to process result of fulfillment
Process callbacks in the order they are registered with each being passed the result of the predecessor
Can be quickly created for static data with defer.sucess and defer.failure functions
Callback Example
d = defer.Deferred()
d.addCallback(lambda x: x + “ is an “)
d.addCallback(lambda x: x + “ example!“)
d.callback(“This”)
>>> This is an example!
Deferred Error Handling
Deferred objects catch all callback Exceptions
Exceptions are wrapped in a Failure object which can also be returned by a callback or errback
If no errbacks remain, exception is swallowed and processing is stopped
Remaining callbacks before next errback are skipped
If errback returns non-Failure, callback processing resumes
Callback Non-Failure Flow
Deferred callback(result)• callback - success• callback – success• errback - skipped• errback – skipped
Errback Non-Failure Flow
Deferred callback(result)• callback - success• errback - failure• callback – success• errback – skipped
Errback Failure Flow
Deferred callback(result)• callback - success• callback – failure• errback - failure• errback – success
Threads = Concurrent
Threads run in thread pool managed by reactor
callInThread executes function in its own thread with supplied data
deferToThread works like callInThread but returns a Deferred object and processes the callbacks when complete
Use threads for blocking code
Reactor
Manages the main event loop
Waits for and dispatches events or messages
May have predefined services to provide access to network communications, threading, and event dispatching. Available services are operating system dependent
Application Hierarchy
Protocols
Service
Application Mail
SSL
Web Mail SMTP
UDP
StatsD
Application
Starts and stops the reactor
Manages services
Handles logging
Can run as daemon
Service
Registers with the reactor
Processed in the main event loop
Manages connections
Communicates via Protocols
Predefined Services
File Descriptor
Process
Threads
Time (scheduling)
SSL
TCP
UDP
UNIX (UNIX sockets)
Socket (3rd Party Sockets)
Protocol
Utilized for interaction with services
Services with stateful connections utilize protocol factories to generate a protocol instance per connection.
Provided with method to send data to connection in the form of transports or producers
Protocol Factory
Builds protocol instances
Injected into every protocol instance
Meant to perform inter-instance interaction
Can be used as testable registry for registry pattern implementations
Included Protocols
AMP
Basic
Dict
Finger
FTP
GPS
HTB
HTTP
Ident
Loopback
Memcache
Mice
PCP
Portforward
Postfix
Shoutcast
SIP
SOCKSv4
Stateful
Telnet
TLS
Wire
Available Protocols
Mongo
LDAP
Dict
Finger
FTP
GPS
HTB
HTTP
Ident
Loopback
Memcache
Mice
PCP
Portforward
Postfix
Shoutcast
SIP
SOCKSv4
Stateful
Telnet
TLS
Wire
MongoDB*
LDAP*
* External packages maintained by Twisted Matrix Labs
Twisted Web
Web server built on Twisted HTTP Protocol
Can be stand alone web server
Can embedded as part of an application
Supports all of the features of most modern web servers
Web Server Features
Virtual hosts
Proxy/Reverse proxy
Scripts (hello.py)
Static files
CGI
WSGI
URL rewrites
MIME processors
Sessions
Simple Web Application
Site object is root Resource
Resource object represents URL segment
Tree hierarchy of resources
Can directly return string or write response via request object for deferred processing
Example Hierarchy
/foo/
bar/
bat/fizz/buz
Example Hierarchy Code
root = RootResource()foo = FooResource()foo.putChild(“bar”, BarResource())foo.putChild(“bat”, BatResource())root.putChild(“foo”, foo)root.putChild(“fiz/buzz”, FizBuzzResource())reactor.listenTCP(8080, Site(root))reactor.start()
Concurrent Example
from twisted.web import resource
class DemoResource(resource.Resource):
def render_GET(self, request):
return "<html>foo</html>"
Deferred Exampledef render_GET(self, request):
def process_resp(html, request):
request.write(html)
request.finish()
deferred = self.async_client.request()
deferred.addCallback(process_resp, request)
return NOT_DONE_YET
Demo Time
Example on GitHub: https://github.com/aenglander/demo-python-twisted-websocket
@adam_englander
#launchkey on freenode.net
#vegastech on freenode.net