Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Working with Asynchronous Events

298 vues

Publié le

Talk given at the PHP user group Frankfurt regarding asynchronous events in Symfony 2 and the RabbitMQ library. We demonstrate some ups and downs as well as best practices and lessons learned.

Publié dans : Logiciels
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Working with Asynchronous Events

  1. 1. Working with Asynchronous Events Jan Gregor Emge-Triebel (@jan0707) Dennis Oehme (@dennisoehme)
  2. 2. Garden of Concepts GmbH ● Hanau / Munich ● In house CMS “.stone” ● In house CRM “Elvis” ● Various individual customer projects ● Symfony based development, supported by: ○ MongoDB, ElasticSearch, MySQL, RabbitMQ, Redis, AngularJS, Twig, Jade, Gitlab, Docker, Vagrant, Ansible ● Say hi @goc_mediastudio
  3. 3. Requests == Event “Symfony is a request based framework.” Fabien Potencier, Symfony Live 2014 @fabpot
  4. 4. Requests Request Controller Logic 1 Logic 2 Response Request Controller Logic 1 Logic 2 Response Synchronous Asynchronous
  5. 5. Why do we need asynchronous event processing ? ● Move long running logic ● Cut down reaction times ● Transfer load ● Microservices ● Exchange data/messages with other (sub-)systems ● Parallelization ● Scaling
  6. 6. Our journey ● Gearman ● (PHP) Resque ○ Demo ● RabbitMQ
  7. 7. Gearman ● Gearman Job Server (written in C) ● Runs as a Linux daemon ● Requires PECL extension for worker API ● Worker APIs exists for other Languages (C, Perl,...)
  8. 8. Gearman http://gearman.org/examples/send-emails/
  9. 9. Gearman ● Latest PECL-Extension version is 1.1.2 (August 2013) ● Latest Server-Release (spring 2014) ● No more process is being made ● Gearman daemon and extension are often incompatible
  10. 10. (PHP) Resque ● Is a Resque port to php: ○ Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. ● Needs Redis(-Cluster) ● Redis is capable of atomic transactions (FIFO)
  11. 11. Demo https://github.com/Jan0707/async-symfony
  12. 12. (PHP) Resque ● Hardly any commits since 2012 ● Last commit to Master on 2015-05-12 ● chrisboulton/php-resque & michelsalib/BCCResqueBundle must be presumed dead ?
  13. 13. RabbitMQ Or why this talk should be named: “Working with Asynchronous Events in RabbitMQ”
  14. 14. RabbitMQ ● Message Broker written in Erlang ● AMQP (Advanced Message Queuing Protocol) ● Open, but binary network protocoll ● Routing, message distribution ● Transactions ● Plus, more nice features ○ TTL ○ DLX (Dead Letter eXchange) ○ PreFetching
  15. 15. Of publishers and consumers ... Publisher Publish Consumer Ex- change QueueRoutes Consumes
  16. 16. There is a Bundle for it ● https://github.com/php-amqplib/RabbitMqBundle ● Consumer and Producer as Symfony services ● Every queue Consumer runs as a Symfony command ● Configuration via app/config.yml ○ Better: Create app/config/rabbitmq.yml and import in app/config/config.yml
  17. 17. app/config/rabbitmq.yml
  18. 18. Symfony Producer
  19. 19. Symfony Consumer
  20. 20. Lessons Learned: Consumer ● PHP threads are not meant to be long running ○ New problems: memory leaks, ... ● Use supervisor to check and restart Consumers ● Open database connections, timeouts ○ Better quit when inactive ● Cap maximum runtime per thread ○ Automatically shutdown after processign x messages ● Memory limit: RabbitMQ-Bundle utilizes “soft-quota” ● Booting consumers takes rather long ● Prefetching (Warning: Order!)
  21. 21. Lessons Learned: Message ● Compact/short messages ● Queues eat up a lot of storage / RAM ● Common format such as JSON or XML ○ Do not serialize PHP objects or arrays ○ Ensure interoperability with other systems and languages
  22. 22. Lessons Learned: Message Transformation ● Use JMS/Serializer to convert objects to JSON ○ Deserialize JSON to PHP objects in consumer ○ Objects can and should be validated (with symfony forms and asserts) ○ Reload entities / documents to keep them up to date (doctrine/unit of work) ● Alternative approach: PHPs JsonSerializable Interface
  23. 23. Lessons Learned: Message Header ● Use message “Header” for content-type and version ○ Message formats can change ○ consumers must remain compatible ● Unqiue IDs make logging / debugging / live easier ● Other meta data: ○ timestamp ○ user / session ○ App id ○ ...
  24. 24. Tip: Custom Consumer-/Producer-Classes ● Message transformation ● Events ● Logging
  25. 25. Transactions ● Messages must be brokered to exactly one Consumer ● Consumer must explicitly answer a message with “acknowledge” ● If not, the message will be marked “rejected” and be requeued ○ Beware of requeuing: ■ Can lead to toxic messages ■ Can change the order of messages ○ Beware of final rejection: ■ Can lead to loss of data
  26. 26. Toxic Messages ● If a message is not explicitly being answered, it will be requeued. ● If that happens multiple times : ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop
  27. 27. Dead Letter Exchange ● RabbitMQ comes with a Dead Letter Exchange (DLX) ● Every queue determines its own or shared DLX ● DLX is a separate queue, storing all rejected messages ○ (Reject, TTL, Prefetch)
  28. 28. app/config/rabbitmq.yml
  29. 29. Signal Handling ● Graceful Shutdown ● pcntl_signal / _dispatch ● SIGTERM ● FATAL ERROR abfangen ● Logging ● Toxic Message-Handling Shutdown
  30. 30. Deployment ● Consumers need an up to date code base ● Consumers need a graceful shutdown ● Consumers must only be shutdown when idling ○ Otherwise : Loss of message / data
  31. 31. Other challenges ● Significant change of infrastructure ○ Choose your libraries and dependencies carefully! ● Moving load is not removing load ● Rethinking necessary ○ Display of status / intermediate result ○ Websockets or Push instead of Polling ● Interdependencies of messages ○ Especially when running parallel consumers! ● Performance (Doctrine ORM / ODM)
  32. 32. Questions ? Thank you :) Find this and other talks at: https://www.gardenofconcepts. com/talks/ Feedback is always welcome & please rate our talk !

×