introduction to asyncio

@saghul

Introduction to asyncioSaúl Ibarra Corretgé

PyLadies Amsterdam - 20th March 2014

github.com/saghul

Sockets 101

import socket

server = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)server.bind(('127.0.0.1', 1234))server.listen(128)print("Server listening on: {}".format(server.getsockname()))

client, addr = server.accept()print("Client connected: {}".format(addr))

while True: data = client.recv(4096) if not data: print("Client has disconnected") break client.send(data)

server.close()

Scaling Up!

import socketimport _thread

def handle_client(client, addr): print("Client connected: {}".format(addr)) while True: data = client.recv(4096) if not data: print("Client has disconnected") break client.send(data.upper())

server = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)server.bind(('127.0.0.1', 1234))server.listen(128)print("Server listening on: {}".format(server.getsockname()))

while True: client, addr = server.accept() _thread.start_new_thread(handle_client, (client, addr))

Scaling Up! (really?)

Threads are too expensive

Also, context switching

Use an event loop instead!

The Async Way (TM)

Single thread

Block waiting for sockets to be ready to read or write

Perform i/o operations and call the callbacks!

Repeat

(Windows is not like this)

Why asyncio?

asyncore and asynchat are not enough

Fresh new implementation of Asynchronous I/O

Python >= 3.3

Trollius: backport for Python >= 2.6

Use new language features: yield from

Designed to interoperate with other frameworks

Components

Event loop, policy

Coroutines, Futures, Tasks

Transports, Protocols

asyncio 101

import asyncio

loop = asyncio.get_event_loop()

@asyncio.coroutinedef infinity(): while True: print("hello!") yield from asyncio.sleep(1)

loop.run_until_complete(infinity())

Coroutines, futures & tasks

Coroutines, futures & tasksCoroutine

generator function, can receive values

decorated with @coroutine

Future

promise of a result or an error

Task

Future which runs a coroutine

FuturesSimilar to Futures from PEP-3148

concurrent.futures.Future

API (almost) identical:

f.set_result(); r = f.result()

f.set_exception(e); e = f.exception()

f.done(); f.cancel(); f.cancelled()

f.add_done_callback(x); f.remove_done_callback(x)

Futures + Coroutines

yield from works with Futures!

f = Future()

Someone will set the result or exception

r = yield from f

Waits until done and returns f.result()

Usually returned by functions

Undoing callbacks

@asyncio.coroutinedef sync_looking_function(*args): fut = asyncio.Future() def cb(result, error): if error is not None: fut.set_result(result) else: fut.set_exception(Exception(error)) async_function(cb, *args) return (yield from fut)

Tasks

Unicorns covered in fairy dust

It’s a coroutine wrapped in a Future

WAT

Inherits from Future

Works with yield from!

r = yield from Task(coro(...))

Tasks vs coroutines

A coroutine doesn’t “advance” without a scheduling mechanism

Tasks can advance on their own

The event loop is the scheduler!

Magic!

Echo Serverimport asyncio

loop = asyncio.get_event_loop()

class EchoProtocol(asyncio.Protocol): def connection_made(self, transport): print('Client connected') self.transport = transport def data_received(self, data): print('Received data:',data) self.transport.write(data) def connection_lost(self, exc): print('Connection closed', exc)

f = loop.create_server(lambda: EchoProtocol(), 'localhost', 1234)server = loop.run_until_complete(f)print('Server started')

loop.run_forever()

Echo Server Reloadedimport asyncio

loop = asyncio.get_event_loop()clients = {} # task -> (reader, writer)

def accept_client(client_reader, client_writer): task = asyncio.Task(handle_client(client_reader, client_writer)) clients[task] = (client_reader, client_writer) def client_done(task): del clients[task] task.add_done_callback(client_done)

@asyncio.coroutinedef handle_client(client_reader, client_writer): while True: data = (yield from client_reader.readline()) client_writer.write(data)

f = asyncio.start_server(accept_client, '127.0.0.1', 1234)server = loop.run_until_complete(f)loop.run_forever()

HTTP Serverimport asyncioimport aiohttpimport aiohttp.server

class HttpServer(aiohttp.server.ServerHttpProtocol):

@asyncio.coroutine def handle_request(self, message, payload): print('method = {!r}; path = {!r}; version = {!r}'.format( message.method, message.path, message.version)) response = aiohttp.Response(self.transport, 200, close=True) response.add_header('Content-type', 'text/plain') response.send_headers() response.write(b'Hello world!\r\n') response.write_eof()

loop = asyncio.get_event_loop()f = loop.create_server(lambda: HttpServer(), 'localhost', 1234)server = loop.run_until_complete(f)loop.run_forever()

Redis

import asyncioimport asyncio_redis

@asyncio.coroutinedef subscriber(channels): # Create connection connection = yield from asyncio_redis.Connection.create(host='localhost', port=6379) # Create subscriber. subscriber = yield from connection.start_subscribe() # Subscribe to channel. yield from subscriber.subscribe(channels) # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel)

loop = asyncio.get_event_loop()loop.run_until_complete(subscriber(['my-channel']))

More?

We just scratched the surface!

Read PEP-3156 (it’s an easy read, I promise!)

Checkout the documentation

Checkout the third-party libraries

Go implement something cool!

“I hear and I forget. I see and I remember.

I do and I understand.” - Confucius

Questions?

@saghulbettercallsaghul.com