async web and python
TRANSCRIPT
Jasim Muhammed
ASYNC WEB AND PYTHON
ME?• Senior Software Architect,
Fetchr, Dubai• Hacking around PostgreSQL,
NoSQL, Python, Go, C
SINGLE DELIVERY GUY SHOP
Pic source: https://www.nginx.com/blog/thread-pools-boost-performance-9x/
ASYNC DELIVERY AT SHOP
Pic source: https://www.nginx.com/blog/thread-pools-boost-performance-9x/
HOW REQUEST IS PROCESSED?
Browser
APACHE REQUEST MODES
• Prefork
• Worker
• Event
PYTHON CONCURRENCY OPTIONS
• Multiprocessing
• Threading
• Async processes
PYTHON CONCURRENCY OPTIONSTheory Reality
Source: https://www.reddit.com/r/aww/comments/2oagj8/multithreaded_programming_theory_and_practice
ISSUES WITH MULTIPROCESSING
• Time to create process
• High processor usage
• Private memory space for processes
ISSUES WITH THREADING
• Python threads are run in serial due to GIL
• High processor usage
ASYNC ADVANTAGES
• Process request when an IO bound operation happens
• No high processor usage
• Moderate operations
ASYNC LIBRARIES
• Gevent (Python 2.7)
• Asyncio(Python 3)
DISTRIBUTED JOB SERVERCELERY
• Offload heavy processing
• Distribute processing to multiple servers
• Cron or maintenance jobs
• Easier to implement Webhooks
HOW CELERY HELPS INCONCURRENT REQUESTS?
• Request processing is done very fast
• Offloads heavy computation or third party request services to job server workers.
• Releases the application server workers for further requests
CELERY TASK EXAMPLE• Regular Async works without any returns
• Mail sending
• SMS Sending
• In case Task computation result is important
• Report generation
• Cron/Maintenance jobs like temporary files cleanup
BEST PRACTISES• Separate queue for separate tasks
• Run workers to multiple nodes
• Task retry is there, but be careful. In the case of task failed, you can push to new queue
• Don’t restart workers with ‘KILL -9’, use ‘KILL -15’ or SIGTERM
• Pass only json serialisable data from to celery.
• Use RabbitMQ for broker and Redis for result backend
• AsyncDelay and JobID
