the dos and don’ts of task queues - pretalx
TRANSCRIPT
![Page 1: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/1.jpg)
The dos and don’ts of task queues
PyCon UK 2019
Petr Stehlík@petrstehlik
![Page 2: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/2.jpg)
Petr Stehlík
Python engineer @ Kiwi.com
Finance tribe
$ whoami
![Page 3: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/3.jpg)
1. Task queues2. The story3. Examples vs. reality4. Final setup5. How we do it in Kiwi.com6. Lessons learned7. Q&A
Outline
![Page 4: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/4.jpg)
Task queues
![Page 5: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/5.jpg)
“parallel execution of discrete tasks without blocking”
● Not just Celery
● Major parts
○ Queue
○ Task – unit of work
○ Producer
○ Consumer
What is a task queue
![Page 6: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/6.jpg)
● Decouple long-running task from a synchronous call
● Perform something periodically
● Break down software to more isolated pieces (when microservice is
too big)
● Minimize wait time, latency and/or response time
● Increase throughput of the system
For what is a task queue
![Page 7: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/7.jpg)
The story
![Page 8: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/8.jpg)
The story
![Page 9: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/9.jpg)
The story
![Page 10: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/10.jpg)
“New is always better.”
The story
![Page 11: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/11.jpg)
“Think outside the box.”
The story
![Page 12: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/12.jpg)
“I know everything I need.”
The story
![Page 13: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/13.jpg)
“I can do it better.”
The story
![Page 14: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/14.jpg)
Examples vs. realitywhy it all happened
![Page 15: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/15.jpg)
Example Celery/RQ
![Page 16: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/16.jpg)
Reality RQ
![Page 17: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/17.jpg)
Reality Celery
![Page 18: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/18.jpg)
Final setup
![Page 19: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/19.jpg)
● Python + PostgreSQL
● Flask
● Connexion
● Celery
● Redis on AWS
● Multiple deploy targets
● Logz.io & Datadog
● Sentry
● PagerDuty
Final setup
![Page 20: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/20.jpg)
How we do it in Kiwi.comIn finance tribe
![Page 21: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/21.jpg)
● Python + PostgreSQL
● Flask/AioHttp
● Connexion
● Celery
● Redis on AWS
● Multiple deploy targets
● Logz.io & Datadog
● Sentry
● PagerDuty
Kiwi.com | Finance Tribe toolset
![Page 22: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/22.jpg)
● Python
○ New projects always 3.6+
○ Old projects transitioning from 2.7 to 3.6
○ Monolith -> microservice architecture
● Flask/AioHttp
○ Our go-to framework
○ Boilerplates
○ Quick scaffolding
Kiwi.com | Finance Tribe toolset
![Page 23: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/23.jpg)
● Connexion
○ OpenAPI 3
○ Token-based authentication & authorization
● Celery
○ Follow the best practices (next section)
● Redis on AWS
○ Reliability
○ Easy to deploy
Kiwi.com | Finance Tribe toolset
![Page 24: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/24.jpg)
● Multiple deploy targets
○ HTTP API, Workers, etc.
○ Internal tool for deploying from Gitlab CI
● Logz.io & Datadog
○ Extensive logging
● Sentry
○ When something goes wrong
● PagerDuty
○ When something goes really wrong
Kiwi.com toolset | Finance Tribe
![Page 25: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/25.jpg)
Lessons learned
![Page 26: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/26.jpg)
Lessons learned
Use Redis or AMQP broker (never a database)
![Page 27: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/27.jpg)
Lessons learned
Pass simple objects to the tasks
![Page 28: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/28.jpg)
Lessons learned
Do not wait for tasks inside tasks
![Page 29: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/29.jpg)
Lessons learned
Set retry limit
![Page 30: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/30.jpg)
Lessons learned
Use autoretry_for
![Page 31: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/31.jpg)
Lessons learned
Use retry_backoff=True and retry_jitter=True
![Page 32: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/32.jpg)
Lessons learned
Set hard and soft time limits
![Page 33: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/33.jpg)
Lessons learned
Use bind for a bit of extra oomph (logs, handling, etc.)
![Page 34: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/34.jpg)
Lessons learned
Use separate queues for demanding tasks (set priorities)
![Page 35: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/35.jpg)
Lessons learned
Prefer idempotency and atomicity
"Idempotence is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application."
- Wikipedia
“Atomic operation appears to the rest of the system to occur instantaneously. Atomicity is a guarantee of isolation from concurrent processes.
- Wikipedia
![Page 36: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/36.jpg)
● Use Redis or AMQP (RabbitMQ) broker (never a database)
● Pass simple objects to the tasks
● Do not wait for tasks inside tasks
● Set retry limit
● Use autoretry_for
● Use retry_backoff=True and retry_jitter=True
● Set hard and soft time limits
● Use bind for a bit of extra oomph in tasks (logging, handling, etc.)
● Use separate queues for demanding tasks (set priorities)
● Prefer idempotency and atomicity
Lessons learned
![Page 37: The dos and don’ts of task queues - pretalx](https://reader036.vdocument.in/reader036/viewer/2022062222/62a4715bab5dd42482328239/html5/thumbnails/37.jpg)
● Sharing codebase between producer and consumer (producer must know
everything about consumer and vica versa)
● Use celery to its full potential -> read celery’s docs
● Scalability of 3rd party APIs
Things to consider