1. A Jobs Queue for processing tasks
asynchronously
Guewen Baconnier & Leonardo Pistone
Camptocamp
1 / 32

2. Guewen Baconnier
Developer @ Camptocamp
OCA committer, OCA Delegate
Connector author
@guewenb
@guewen
Leonardo Pistone
Developer @ Camptocamp
OCA committer, OCA Delegate
@lepistone
About us
2 / 32

3. Computers are slow!
3 / 32

4. Computers are slow!
Humans want them to be fast!
3 / 32

5. The problem
4 / 32

6. User Odoo Server
Waiting
Heavytask!
5 / 32

7. Loading...
6 / 32

8. Still loading...
7 / 32

9. Still loading... Please be patient.
8 / 32

10. Don't leave yet, it's still loading
9 / 32

11. You may not believe it, but the application is
actually loading...
10 / 32

12. Take a minute to get a coffee, because it's
loading...
11 / 32

13. Come on...
12 / 32

14. We can try to save a few seconds
13 / 32

15. We can try to save a few seconds
But we have more radical solutions
13 / 32

16. User Odoo Server
Heavytask!
Connector
Runner
14 / 32

17. Connector
odoo-connector.com
15 / 32

18. Queue it!
Dependency on connector
16 / 32

19. Queue it!
Dependency on connector
Declare a job:
fromopenerp.addons.connector.queue.jobimportjob
@job
defa_heavy_task(session,model_name,record_id):
#doanheavytaskonrecord_idofmodel_name
16 / 32

20. Queue it!
Dependency on connector
Declare a job:
fromopenerp.addons.connector.queue.jobimportjob
@job
defa_heavy_task(session,model_name,record_id):
#doanheavytaskonrecord_idofmodel_name
Delay a job:
session=ConnectorSession.from_env(self.env)
a_heavy_task.delay(session,'res.partner',1)
16 / 32

21. Dequeue it!
Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4
17 / 32

22. Dequeue it!
Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4
ODOO_CONNECTOR_CHANNELS=root:3,root.csv:1,root.magento:3
./openerp-server--load=web,connector--workers=4
17 / 32

23. Channels
18 / 32

24. HTTPWorkers
csv
magento
root
Running jobs
Capacity
Channels
19 / 32

25. Properties
20 / 32

26. Priority
10 50 999
Priority
import_order.delay(session,1) #defaultis10
import_order.delay(session,2,priority=50)
import_order.delay(session,10,priority=999)
21 / 32

27. B C
Now + 6:00 + 12:00 +18:00 + 24:00
A
ETA
import_order.delay(session,1) #A
import_order.delay(session,1,eta=6*60*60) #B
import_order.delay(session,2,eta=datetime.now()+timedelta(days=1)) #C
22 / 32

28. Retries
import_order.delay(session,1,max_retries=3)
23 / 32

29. Retries
import_order.delay(session,1,max_retries=3)
Invoke a retry
@job
defimport_order(session,args):
try:
do_operation()
except(socket.gaierror,socket.error,socket.timeout)aserr:
raiseRetryableError(
'Anetworkerrorcausedthefailureofthejob:'
'%s'%err)
23 / 32

30. Best Practices
24 / 32

31. Outdating Data in jobs can become outdated.
No:
@job
defexample(session,record_id,vals):
export(record_id,vals)
Yes:
@job
defexample(session,record_id):
export(session.env['model'].browse(record_id))
25 / 32

32. Outdating
Existence
A job can refer to a record which has been deleted. Always
check if it still exists.
No:
@job
defexample(session,record_id):
export(session.env['model'].browse(record_id))
Yes:
@job
defexample(session,record_id):
record=session.env['model'].browse(record_id)
ifrecord.exists():
export(record)
26 / 32

33. Outdating
Existence
Idempotence
A job should, when possible, produce the same result
when executed several times.
No:
@job
defexample(session,record_id):
export(session.env['model'].browse(record_id))
Yes:
@job
defexample(session,record_id):
record=session.env['model'].browse(record_id)
ifrecord.exists():
ifnotrecord.exported:
export(session.env['model'].browse(record_id))
27 / 32

34. Useful Patterns
28 / 32

35. Fanout Job
A job generating other jobs.
@job
defimport_file(session,filepath):
withopen(filepath)asf:
forlineinf:
import_line.delay(session,line)
29 / 32

36. Try or delay
If an operation failed, try it later.
@job
defdo_operation(session,args):
#work
try:
do_operation(session,args)
exceptTimeoutError:
do_operation.delay(session,args,eta=10*60)
30 / 32

37. Extract highly concurrent tasks
And put them in a one-by-one channel.
31 / 32

38. Thanks!
32 / 32

39. Thanks!
OCA Sponsors
32 / 32