celery for internal api in soa infrastructure
DESCRIPTION
Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)TRANSCRIPT
Celery for internal API in SOA infrastructure
Roman ImankulovPyCon Russia, Feb 2013
1. Internal API. What's the deal
SOA service architecture example
1. Internal API. What's the deal
● DCOM: too Microsoftish
● CORBA: too Enterprisy
● SOAP: XML inside
● RESTful: too sloppy
Celery to the rescue?
Agenda
1. Internal API. What's the deal
2. How to use Celery for internal API
3. Organizing Celery-based API. Routing
4. Celery-based API. Benefits, pitfalls and
security notes
2. How to use Celery for internal API
Celery Worker
2. How to use Celery for internal API
Classic celery client imports task function
2. How to use Celery for internal API
Classic celery client imports makes you:● install everything
○ slows down installation
● import everything○ python processes grow in size○ circular dependencies○ tight coupling
● share everything○ ... with outsource developers
2. How to use Celery for internal API
"Task implementation is not to be exposed without necessity"
William of Ockham
2. How to use Celery for internal API
Low-level interface defines only task name
2. How to use Celery for internal API
Convenient API requires some boilerplate code
2. How to use Celery for internal API
Make it even more convenient with introspection
2. How to use Celery for internal API
Introspection magic with celery-api
http://github.com/imankulov/celery-api
3. Celery-based API. Routing
3. Celery-based API. Routing
Creating API endpoint
1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with
CELERY_IMPORTS
4. Launch worker withcelery worker -A module_name -Q foo
-n foo
3. Celery-based API. Routing
Send tasks with
add.apply_async(queue='foo', args=(1, 1))
3. Celery-based API. Routing
Define routing rules
3. Celery-based API. Routing
Task in the wire
{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
3. Celery-based API. Routing
Task in the wire
"delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
3. Celery-based API. Routing
AMQP for dummies
3. Celery-based API. Routing
Define all queues (API endpoints) you have
4. Celery-based API. Benefits
High level protocol.
Takes the responsibility for
data serialization routingexception handling
4. Celery-based API. Benefits
Support for wide range of brokers.
You can switch between brokers with no major changes in your code
Brokers:RabbitMQ Redis SQLAlchemy Django MongoDB Amazon SQSCouchDB Beanstalk
4. Celery-based API. Benefits
Built-in extra stuff which you were afraid to ever dream of.
asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed tasks limiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)
4. Celery-based API. Pitfalls
Python-centric.
Celery is written in Python and meant to be used with Python code exclusively
4. Celery-based API. Pitfalls
Increasing complexity.
Celery codebase 46k LOCKombu codebase 19k LOC
CompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC
4. Celery-based API. Security note
Never ever expose Celery broker to the Web!
4. Celery-based API. Security note
1. Try >>> redis_url = 'redis://:[email protected]'>>> celery = Celery(broker=redis_url, result=redis_url)
2. Get the contents of /etc/pycon_secret
3. Exchange it for a secret prize ;)