a jobs queue for processing tasks asynchronously
TRANSCRIPT
A Jobs Queue for processing tasksasynchronously
Guewen Baconnier & Leonardo Pistone
Camptocamp
1 / 32
Guewen Baconnier
Developer @ CamptocampOCA committer, OCA DelegateConnector author
@guewenb
@guewen
Leonardo Pistone
Developer @ CamptocampOCA committer, OCA Delegate
@lepistone
About us
2 / 32
Computers are slow!
3 / 32
Computers are slow!
Humans want them to be fast!
3 / 32
The problem
4 / 32
User Odoo Server
Wai
ting
Hea
vy t
ask!
5 / 32
Loading...
6 / 32
Still loading...
7 / 32
Still loading... Please be patient.
8 / 32
Don't leave yet, it's still loading
9 / 32
You may not believe it, but the application isactually loading...
10 / 32
Take a minute to get a coffee, because it'sloading...
11 / 32
Come on...
12 / 32
We can try to save a few seconds
13 / 32
We can try to save a few seconds
But we have more radical solutions
13 / 32
User Odoo Server
Hea
vy t
ask!
Connector Runner
14 / 32
Connectorodoo-connector.com
15 / 32
Queue it!
Dependency on connector
16 / 32
Queue it!
Dependency on connectorDeclare a job:
from openerp.addons.connector.queue.job import job
@jobdef a_heavy_task(session, model_name, record_id): # do an heavy task on record_id of model_name
16 / 32
Queue it!
Dependency on connectorDeclare a job:
from openerp.addons.connector.queue.job import job
@jobdef a_heavy_task(session, model_name, record_id): # do an heavy task on record_id of model_name
Delay a job:
session = ConnectorSession.from_env(self.env)a_heavy_task.delay(session, 'res.partner', 1)
16 / 32
Dequeue it!Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2 ./openerp-server --load=web,connector --workers=4
17 / 32
Dequeue it!Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2 ./openerp-server --load=web,connector --workers=4
ODOO_CONNECTOR_CHANNELS=root:3,root.csv:1,root.magento:3 \ ./openerp-server --load=web,connector --workers=4
17 / 32
Channels
18 / 32
HTT
P W
ork
ers
csv
magento
root
Running jobsCapacity
Channels
19 / 32
Properties
20 / 32
Priority
10 50 999
Priorityimport_order.delay(session, 1) # default is 10import_order.delay(session, 2, priority=50)import_order.delay(session, 10, priority=999)
21 / 32
B C
Now + 6:00 + 12:00 +18:00 + 24:00
A
ETAimport_order.delay(session, 1) # Aimport_order.delay(session, 1, eta=6*60*60) # Bimport_order.delay(session, 2, eta=datetime.now() + timedelta(days=1)) # C
22 / 32
Retriesimport_order.delay(session, 1, max_retries=3)
23 / 32
Retriesimport_order.delay(session, 1, max_retries=3)
Invoke a retry@jobdef import_order(session, args): try: do_operation() except (socket.gaierror, socket.error, socket.timeout) as err: raise RetryableError( 'A network error caused the failure of the job: ' '%s' % err)
23 / 32
Best Practices
24 / 32
Outdating Data in jobs can become outdated.
No:
@jobdef example(session, record_id, vals): export(record_id, vals)
Yes:
@job def example(session, record_id): export(session.env['model'].browse(record_id))
25 / 32
Outdating
Existence
A job can refer to a record which has been deleted. Alwayscheck if it still exists.
No:
@jobdef example(session, record_id): export(session.env['model'].browse(record_id))
Yes:
@job def example(session, record_id): record = session.env['model'].browse(record_id) if record.exists(): export(record)
26 / 32
Outdating
Existence
Idempotence
A job should, when possible, produce the same resultwhen executed several times.
No:
@jobdef example(session, record_id): export(session.env['model'].browse(record_id))
Yes:
@job def example(session, record_id): record = session.env['model'].browse(record_id) if record.exists(): if not record.exported: export(session.env['model'].browse(record_id))
27 / 32
Useful Patterns
28 / 32
Fanout JobA job generating other jobs.
@jobdef import_file(session, filepath): with open(filepath) as f: for line in f: import_line.delay(session, line)
29 / 32
Try or delayIf an operation failed, try it later.
@jobdef do_operation(session, args): # work
try: do_operation(session, args)except TimeoutError: do_operation.delay(session, args, eta=10*60)
30 / 32
Extract highly concurrent tasksAnd put them in a one-by-one channel.
31 / 32
Thanks!
32 / 32
Thanks!
OCA Sponsors
32 / 32