A Jobs Queue for processing tasks
asynchronously
Guewen Baconnier & Leonardo Pistone
Camptocamp
1 / 32
Guewen Baconnier
Developer @ Camptocamp
OCA committer, OCA Delegate
Connector author
@guewenb
@guewen
Leonardo Pistone
Dev...
Computers are slow!
3 / 32
Computers are slow!
Humans want them to be fast!
3 / 32
The problem
4 / 32
User Odoo Server
Waiting
Heavytask!
5 / 32
Loading...
6 / 32
Still loading...
7 / 32
Still loading... Please be patient.
8 / 32
Don't leave yet, it's still loading
9 / 32
You may not believe it, but the application is
actually loading...
10 / 32
Take a minute to get a coffee, because it's
loading...
11 / 32
Come on...
12 / 32
We can try to save a few seconds
13 / 32
We can try to save a few seconds
But we have more radical solutions
13 / 32
User Odoo Server
Heavytask!
Connector
Runner
14 / 32
Connector
odoo-connector.com
15 / 32
Queue it!
Dependency on connector
16 / 32
Queue it!
Dependency on connector
Declare a job:
fromopenerp.addons.connector.queue.jobimportjob
@job
defa_heavy_task(sess...
Queue it!
Dependency on connector
Declare a job:
fromopenerp.addons.connector.queue.jobimportjob
@job
defa_heavy_task(sess...
Dequeue it!
Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4
17 / 32
Dequeue it!
Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4
ODOO_CONN...
Channels
18 / 32
HTTPWorkers
csv
magento
root
Running jobs
Capacity
Channels
19 / 32
Properties
20 / 32
Priority
10 50 999
Priority
import_order.delay(session,1) #defaultis10
import_order.delay(session,2,priority=50)
import_or...
B C
Now + 6:00 + 12:00 +18:00 + 24:00
A
ETA
import_order.delay(session,1) #A
import_order.delay(session,1,eta=6*60*60) #B
...
Retries
import_order.delay(session,1,max_retries=3)
23 / 32
Retries
import_order.delay(session,1,max_retries=3)
Invoke a retry
@job
defimport_order(session,args):
try:
do_operation()...
Best Practices
24 / 32
Outdating Data in jobs can become outdated.
No:
@job
defexample(session,record_id,vals):
export(record_id,vals)
Yes:
@job
...
Outdating
Existence
A job can refer to a record which has been deleted. Always
check if it still exists.
No:
@job
defexamp...
Outdating
Existence
Idempotence
A job should, when possible, produce the same result
when executed several times.
No:
@job...
Useful Patterns
28 / 32
Fanout Job
A job generating other jobs.
@job
defimport_file(session,filepath):
withopen(filepath)asf:
forlineinf:
import_l...
Try or delay
If an operation failed, try it later.
@job
defdo_operation(session,args):
#work
try:
do_operation(session,arg...
Extract highly concurrent tasks
And put them in a one-by-one channel.
31 / 32
Thanks!
32 / 32
Thanks!
OCA Sponsors
32 / 32
Prochain SlideShare
Chargement dans…5
×

A jobs queue for processing tasks asynchronously

814 vues

Publié le

This talk is about the "Connector" [1] addon, but focused on the Jobs Queue, which can be used independently of any "connection". We will show, with examples, how we can use this module to delay intensive tasks which would block the users if they were executed synchronously. We'll also see how we can optimize the workload of the jobs using different Job Channels.

[1] http://odoo-connector.com

Publié dans : Technologie
0 commentaire
2 j’aime
Statistiques
Remarques
  • Soyez le premier à commenter

Aucun téléchargement
Vues
Nombre de vues
814
Sur SlideShare
0
Issues des intégrations
0
Intégrations
3
Actions
Partages
0
Téléchargements
7
Commentaires
0
J’aime
2
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

A jobs queue for processing tasks asynchronously

  1. 1. A Jobs Queue for processing tasks asynchronously Guewen Baconnier & Leonardo Pistone Camptocamp 1 / 32
  2. 2. Guewen Baconnier Developer @ Camptocamp OCA committer, OCA Delegate Connector author @guewenb @guewen Leonardo Pistone Developer @ Camptocamp OCA committer, OCA Delegate @lepistone About us 2 / 32
  3. 3. Computers are slow! 3 / 32
  4. 4. Computers are slow! Humans want them to be fast! 3 / 32
  5. 5. The problem 4 / 32
  6. 6. User Odoo Server Waiting Heavytask! 5 / 32
  7. 7. Loading... 6 / 32
  8. 8. Still loading... 7 / 32
  9. 9. Still loading... Please be patient. 8 / 32
  10. 10. Don't leave yet, it's still loading 9 / 32
  11. 11. You may not believe it, but the application is actually loading... 10 / 32
  12. 12. Take a minute to get a coffee, because it's loading... 11 / 32
  13. 13. Come on... 12 / 32
  14. 14. We can try to save a few seconds 13 / 32
  15. 15. We can try to save a few seconds But we have more radical solutions 13 / 32
  16. 16. User Odoo Server Heavytask! Connector Runner 14 / 32
  17. 17. Connector odoo-connector.com 15 / 32
  18. 18. Queue it! Dependency on connector 16 / 32
  19. 19. Queue it! Dependency on connector Declare a job: fromopenerp.addons.connector.queue.jobimportjob @job defa_heavy_task(session,model_name,record_id): #doanheavytaskonrecord_idofmodel_name 16 / 32
  20. 20. Queue it! Dependency on connector Declare a job: fromopenerp.addons.connector.queue.jobimportjob @job defa_heavy_task(session,model_name,record_id): #doanheavytaskonrecord_idofmodel_name Delay a job: session=ConnectorSession.from_env(self.env) a_heavy_task.delay(session,'res.partner',1) 16 / 32
  21. 21. Dequeue it! Start the server with: ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4 17 / 32
  22. 22. Dequeue it! Start the server with: ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4 ODOO_CONNECTOR_CHANNELS=root:3,root.csv:1,root.magento:3 ./openerp-server--load=web,connector--workers=4 17 / 32
  23. 23. Channels 18 / 32
  24. 24. HTTPWorkers csv magento root Running jobs Capacity Channels 19 / 32
  25. 25. Properties 20 / 32
  26. 26. Priority 10 50 999 Priority import_order.delay(session,1) #defaultis10 import_order.delay(session,2,priority=50) import_order.delay(session,10,priority=999) 21 / 32
  27. 27. B C Now + 6:00 + 12:00 +18:00 + 24:00 A ETA import_order.delay(session,1) #A import_order.delay(session,1,eta=6*60*60) #B import_order.delay(session,2,eta=datetime.now()+timedelta(days=1)) #C 22 / 32
  28. 28. Retries import_order.delay(session,1,max_retries=3) 23 / 32
  29. 29. Retries import_order.delay(session,1,max_retries=3) Invoke a retry @job defimport_order(session,args): try: do_operation() except(socket.gaierror,socket.error,socket.timeout)aserr: raiseRetryableError( 'Anetworkerrorcausedthefailureofthejob:' '%s'%err) 23 / 32
  30. 30. Best Practices 24 / 32
  31. 31. Outdating Data in jobs can become outdated. No: @job defexample(session,record_id,vals): export(record_id,vals) Yes: @job defexample(session,record_id): export(session.env['model'].browse(record_id)) 25 / 32
  32. 32. Outdating Existence A job can refer to a record which has been deleted. Always check if it still exists. No: @job defexample(session,record_id): export(session.env['model'].browse(record_id)) Yes: @job defexample(session,record_id): record=session.env['model'].browse(record_id) ifrecord.exists(): export(record) 26 / 32
  33. 33. Outdating Existence Idempotence A job should, when possible, produce the same result when executed several times. No: @job defexample(session,record_id): export(session.env['model'].browse(record_id)) Yes: @job defexample(session,record_id): record=session.env['model'].browse(record_id) ifrecord.exists(): ifnotrecord.exported: export(session.env['model'].browse(record_id)) 27 / 32
  34. 34. Useful Patterns 28 / 32
  35. 35. Fanout Job A job generating other jobs. @job defimport_file(session,filepath): withopen(filepath)asf: forlineinf: import_line.delay(session,line) 29 / 32
  36. 36. Try or delay If an operation failed, try it later. @job defdo_operation(session,args): #work try: do_operation(session,args) exceptTimeoutError: do_operation.delay(session,args,eta=10*60) 30 / 32
  37. 37. Extract highly concurrent tasks And put them in a one-by-one channel. 31 / 32
  38. 38. Thanks! 32 / 32
  39. 39. Thanks! OCA Sponsors 32 / 32

×