Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

A jobs queue for processing tasks asynchronously

1 130 vues

Publié le

This talk is about the "Connector" [1] addon, but focused on the Jobs Queue, which can be used independently of any "connection". We will show, with examples, how we can use this module to delay intensive tasks which would block the users if they were executed synchronously. We'll also see how we can optimize the workload of the jobs using different Job Channels.

[1] http://odoo-connector.com

Publié dans : Technologie
  • Soyez le premier à commenter

A jobs queue for processing tasks asynchronously

  1. 1. A Jobs Queue for processing tasks asynchronously Guewen Baconnier & Leonardo Pistone Camptocamp 1 / 32
  2. 2. Guewen Baconnier Developer @ Camptocamp OCA committer, OCA Delegate Connector author @guewenb @guewen Leonardo Pistone Developer @ Camptocamp OCA committer, OCA Delegate @lepistone About us 2 / 32
  3. 3. Computers are slow! 3 / 32
  4. 4. Computers are slow! Humans want them to be fast! 3 / 32
  5. 5. The problem 4 / 32
  6. 6. User Odoo Server Waiting Heavytask! 5 / 32
  7. 7. Loading... 6 / 32
  8. 8. Still loading... 7 / 32
  9. 9. Still loading... Please be patient. 8 / 32
  10. 10. Don't leave yet, it's still loading 9 / 32
  11. 11. You may not believe it, but the application is actually loading... 10 / 32
  12. 12. Take a minute to get a coffee, because it's loading... 11 / 32
  13. 13. Come on... 12 / 32
  14. 14. We can try to save a few seconds 13 / 32
  15. 15. We can try to save a few seconds But we have more radical solutions 13 / 32
  16. 16. User Odoo Server Heavytask! Connector Runner 14 / 32
  17. 17. Connector odoo-connector.com 15 / 32
  18. 18. Queue it! Dependency on connector 16 / 32
  19. 19. Queue it! Dependency on connector Declare a job: fromopenerp.addons.connector.queue.jobimportjob @job defa_heavy_task(session,model_name,record_id): #doanheavytaskonrecord_idofmodel_name 16 / 32
  20. 20. Queue it! Dependency on connector Declare a job: fromopenerp.addons.connector.queue.jobimportjob @job defa_heavy_task(session,model_name,record_id): #doanheavytaskonrecord_idofmodel_name Delay a job: session=ConnectorSession.from_env(self.env) a_heavy_task.delay(session,'res.partner',1) 16 / 32
  21. 21. Dequeue it! Start the server with: ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4 17 / 32
  22. 22. Dequeue it! Start the server with: ODOO_CONNECTOR_CHANNELS=root:2./openerp-server--load=web,connector--workers=4 ODOO_CONNECTOR_CHANNELS=root:3,root.csv:1,root.magento:3 ./openerp-server--load=web,connector--workers=4 17 / 32
  23. 23. Channels 18 / 32
  24. 24. HTTPWorkers csv magento root Running jobs Capacity Channels 19 / 32
  25. 25. Properties 20 / 32
  26. 26. Priority 10 50 999 Priority import_order.delay(session,1) #defaultis10 import_order.delay(session,2,priority=50) import_order.delay(session,10,priority=999) 21 / 32
  27. 27. B C Now + 6:00 + 12:00 +18:00 + 24:00 A ETA import_order.delay(session,1) #A import_order.delay(session,1,eta=6*60*60) #B import_order.delay(session,2,eta=datetime.now()+timedelta(days=1)) #C 22 / 32
  28. 28. Retries import_order.delay(session,1,max_retries=3) 23 / 32
  29. 29. Retries import_order.delay(session,1,max_retries=3) Invoke a retry @job defimport_order(session,args): try: do_operation() except(socket.gaierror,socket.error,socket.timeout)aserr: raiseRetryableError( 'Anetworkerrorcausedthefailureofthejob:' '%s'%err) 23 / 32
  30. 30. Best Practices 24 / 32
  31. 31. Outdating Data in jobs can become outdated. No: @job defexample(session,record_id,vals): export(record_id,vals) Yes: @job defexample(session,record_id): export(session.env['model'].browse(record_id)) 25 / 32
  32. 32. Outdating Existence A job can refer to a record which has been deleted. Always check if it still exists. No: @job defexample(session,record_id): export(session.env['model'].browse(record_id)) Yes: @job defexample(session,record_id): record=session.env['model'].browse(record_id) ifrecord.exists(): export(record) 26 / 32
  33. 33. Outdating Existence Idempotence A job should, when possible, produce the same result when executed several times. No: @job defexample(session,record_id): export(session.env['model'].browse(record_id)) Yes: @job defexample(session,record_id): record=session.env['model'].browse(record_id) ifrecord.exists(): ifnotrecord.exported: export(session.env['model'].browse(record_id)) 27 / 32
  34. 34. Useful Patterns 28 / 32
  35. 35. Fanout Job A job generating other jobs. @job defimport_file(session,filepath): withopen(filepath)asf: forlineinf: import_line.delay(session,line) 29 / 32
  36. 36. Try or delay If an operation failed, try it later. @job defdo_operation(session,args): #work try: do_operation(session,args) exceptTimeoutError: do_operation.delay(session,args,eta=10*60) 30 / 32
  37. 37. Extract highly concurrent tasks And put them in a one-by-one channel. 31 / 32
  38. 38. Thanks! 32 / 32
  39. 39. Thanks! OCA Sponsors 32 / 32

×