cooking pies with celery

Печём пирожки с Celery

Александр Мокров

О чем доклад

Немного об очередях

Краткий обзор возможностей Celery

Использование на примере виртуальной фабрики пирожков

Очереди

Producer

queue_name

Consumer

Очереди

Producer

Consumer

Consumer

queue_name

Роутинг

Producer

Consumer

Consumer

first_queue

Xsecond_queue

white

white

black

What is task queue?

Celery: Distributed Task QueueCelery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Celery is used in production systems to process millions of tasks a day.

Ask Solem Hoelhttps://github.com/ask

Principal Software Engineer at Robinhood

San Francisco, CA

Staff Engineer, RabbitMQVMwareмай 2011 – май 2013 (2 года 1 месяц)

Open source and consulting work on Celery, RabbitMQ,

Celery Дата начала: май 2009

CeleryKombu (Messaging library for Python) 41991 (22541) lines

Billiard (Python multiprocessing fork with improvements and bugfixes) 19191(13115)

Vine (promise, async, future) 2921

Celery 104296 (37495)

Total 168399(76072)

It supportsBrokers

RabbitMQ, Redis,

MongoDB (exp), ZeroMQ (exp)

CouchDB (exp), SQLAlchemy (exp)

Django ORM (exp), Amazon SQS, (exp)

Concurrency

prefork (multiprocessing),

Eventlet, gevent

threads/single threaded

Result Stores

AMQP, Redis

memcached, MongoDB

SQLAlchemy, Django ORM

Apache Cassandra

Serialization

pickle, json, yaml, msgpack.

zlib, bzip2 compression.

Cryptographic message signing

Applicationfrom celery import Celeryfrom conf import Settingsfrom tasks.bake_pie import BakePie

APP_NAME = 'pie_fabric'

app = Celery(APP_NAME)app.config_from_object(Settings)app.tasks.register(BakePie())

Settingsclass Settings: BROKER_URL = 'amqp://guest:guest@localhost:5672//' CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0'

$ celery -A pie_fabric worker -l INFO

-------------- celery@amokrov v4.0.0rc2 (0today8)

---- **** -----

--- * *** * -- Linux-3.19.0-61-generic-x86_64-with-Ubuntu-14.04-trusty 2016-06-24 15:52:22

-- * - **** ---

- ** ---------- [config]

- ** ---------- .> app: pie_fabric:0x7efc8cd4c588

- ** ---------- .> transport: amqp://guest:**@localhost:5672//

- ** ---------- .> results: redis://127.0.0.1:6379/0

- *** --- * --- .> concurrency: 4 (prefork)

-- ******* ----

--- ***** ----- [queues]

-------------- .> celery exchange=celery(direct) key=celery

[tasks]

. bake_pie

Architecture

Client 1

Client 2

Worker 1Broker

Task Queue 1

Task Queue 2...

Worker 2

Result Backend

send tasks

send tasks

tasks

tasks

task result

task result

get task resultget task result

Фабрика пирожков

Интерфейс

Заказ

Осуществление заказа

Стряпание пирога

Выпекание пирога

Получение ингредиентов для теста

Ингредиенты Заготовка

Пирог

Создание теста

Получение начинки

Workflow

get flour

bake pie

get meat

seal pie

create dough

order pie

get milk

get aggs

The canvas

Wokrflow primitives

● group● chain● chord● map, starmap, chunks

Tasksfrom celery.app.task import Taskfrom celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

class GetIngredient(Task):

name = 'get_ingredients'

def run(self, ingredient_name): logger.info('Getting {}'.format(ingredient_name))

l # код получения ингредиента

return ingredient

Workflowfrom celery.app.task import Taskfrom celery import canvas, signature

class OrderPie(Task): name = 'order_pie' def run(self): get_meat = signature('get_ingredients', args=('meat',)) ... dough_chord = canvas.chord([get_eggs, get_milk, get_flour], create_dough) components_group = canvas.group(dough_chord, get_meat)

return (components_group | seal_pie | bake_pie).delay()

Client side>>> from tasks.order_pie import OrderPie>>> r = OrderPie().delay()>>> r.id'58c5bb47-8fb3-4d4a-b2f5-8899520b5179'>>> r.ready()True>>> r.get()[['091f9ae7-561a-4781-a4d9-47bbcb121360', [['0865f66b-b89d-4ff3-a272-1f6b01d0a11f', None], None]], None]>>> r.backend.get_task_meta(r.get()[0][0]){'task_id': '091f9ae7-561a-4781-a4d9-47bbcb121360', 'children': [], 'traceback': None, 'result': 'baked pie with meat', 'status': 'SUCCESS'}

Routingfrom kombu import Exchange, Queue

class Router: def route_for_task(self, task, *args, **kwargs): route = {'exchange': Settings.EXCHANGE.name, 'routing_key': 'common'} if task in {'bake_pie', 'get_ingredients', 'seal_pie', 'order_pie'}: route['routing_key'] = 'fabric' elif task == 'create_dough': route = {'exchange': 'dough',

'routing_key': 'dough'} return route

Routingclass Settings: EXCHANGE = Exchange('pie_fabric, type='direct') CELERY_ROUTES = (Router(),) CELERY_QUEUES = ( Queue('pie_fabric.common', EXCHANGE, routing_key='common'), Queue('pie_fabric.fabric', EXCHANGE, routing_key='fabric'), )

--- ***** ----- [queues] -------------- .> pie_fabric.common exchange=pie_fabric(direct) key=common .> pie_fabric.fabric exchange=pie_fabric(direct) key=fabric

Second applicationclass Settings: BROKER_URL = 'amqp://guest:guest@localhost:5672//' CELERY_RESULT_BACKEND = 'redis://127.0.0.1:6379/0' EXCHANGE = Exchange('dough', type='direct') CELERY_QUEUES = ( Queue('dough', EXCHANGE, routing_key='dough'), )

app = Celery('dough')app.config_from_object(Settings)app.tasks.register(CreateDough())

$celery -A dough worker -P gevent --concurrency=1000 -l INFO

- ** ---------- [config]- ** ---------- .> app: dough:0x7f850ee22748- ** ---------- .> transport: amqp://guest:**@localhost:5672//- ** ---------- .> results: redis://127.0.0.1:6379/0- *** --- * --- .> concurrency: 1000 (gevent)-- ******* ---- --- ***** ----- [queues] -------------- .> dough exchange=dough(direct) key=dough

[tasks] . create_dough

Geventgevent is a coroutine -based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libev event loop

gevent.monkey – Make the standard library cooperative

def _patch_gevent(): from gevent import monkey, signal as gsignal, version_info

monkey.patch_all()

Pollingdef run(self, *ingredients): logger.info('Ingredients: {}'.format(ingredients)) id = create_dough(ingredients) # Отправка запроса на приготовление теста while True: time.sleep(polling_timeout) if ready(id): # Проверяем готово ли тесто dough = get_dough(id) # Если готово, то забираем

return dough

Calling Tasks

applyExecute this task locally, by blocking until the task returns.

apply_async

Apply tasks asynchronously by sending a message.

delay

Shortcut to send a task message, but does not support execution options.

retry

Retry the task.

OptionsLinking (callbacks/errbacks)link

link_error

ETA and countdowncountdown

eta

Expirationexpires

Retry

retry=True,

retry_policy={

'max_retries': 3,

'interval_start': 0,

'interval_step': 0.2,

'interval_max': 0.2,

})

SerializersCompressionRouting options

Callbacksoptions = { 'link': app.signature('seal_pie'), 'link_error': app.signature('order_error')}for sub_task in options.values(): sub_task.set( **app.amqp.router.route(sub_task.options, sub_task.task, sub_task.args, sub_task.kwargs) )

Periodic Tasks

from celery.schedule import crontab

CELERYBEAT_SCHEDULE = { 'check-every-minute': { 'task': 'check_ingredients', 'schedule':crontab()}

$ celery -A pie_fabric beat

Concurrency

prefork (multiprocessing)

eventlet/gevent

threads/single threaded

Signals

Task Signals

App Signals

Worker Signals

Beat Signals

Eventlet Signals

Logging Signals

Command signals

Deprecated Signals

Worker Signalsceleryd_after_setupceleryd_initworker_initworker_readyworker_process_initworker_process_shutdownworker_shutdown

Ссылкиhttps://www.rabbitmq.com

http://docs.celeryproject.org

Cпасибо за внимание!

Вопросы?