Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
It’s all about eXperience
Gaetano Giunta
Gilles Guirand
Forum PHP
Paris
2015/11/24
Rabbits, indians and...
Symfony meets
q...
Brokers, don’t they work in Wall Street?
Brokers, don’t they work in Wall Street?
Message brokers are a building block of Message oriented middleware.
[wikipedia]
Brokers, don’t they work in Wall Street?
Message brokers are a building block of Message oriented middleware.
[wikipedia]
...
• RabbitMQ
• ActiveMQ
• Apollo
• Kafka
• Kestrel
• Amazon SQS
• ZeroMQ*
• And many more…
Let’s try again: can you name any...
What do they do?
Where do they differ?
• Protocol support (amqp, stomp, …)
• Libraries exist for writing clients using standard protocols
•...
Where do they differ?
• Surviving failures
• Are messages sent before the client is started buffered or lost?
• What happe...
Everything clear so far?
Everything clear so far?
A case study: generating MSOffice docs
The needs
• Produce content in Microsoft Office formats
• Many formats for each doc...
A case study: generating MSOffice docs
Reality bites
• Slooow
• Rest assured that some of the docs will be pretty big
• Un...
A case study: generating MSOffice docs
…and the results are:
• Ugly code: lots of polling, copying of files around, manual...
TextBook scenario for introducing queues
Web
server
Consumer
Rabbit
MQ
Libre
Office
PHP
TextBook scenario for introducing queues
Web
server
Consumer
?
Rabbit
MQ
Libre
Office
PHP
TextBook scenario for introducing queues
Web
server
PHP
!!!
Rabbit
MQ
Libre
Office
PHP
TextBook scenario for introducing queues
Web
server
PHP
!
Rabbit
MQ
Libre
Office
PHP
TextBook scenario for introducing queues
Web
server
Symfony
!!!
Rabbit
MQ
Libre
Office
Symfony
TextBook scenario for introducing queues
Web
server
Symfony
!!!
Rabbit
MQ
Libre
Office
Symfony
The rationale
• Same library handling both sides of the connection
• Easier to troubleshoot
• Sf Console component is love...
There is a bundle for that
oldsound/rabbitmq-bundle
• specific to RabbitMQ
• supports both producers and consumers
php app...
Long lived php processes ?
• Fatal errors
• Memory leaks
• Database connections might become broken
• Is it in your code o...
A new hope
• kaliop/queueingbundle
• kaliop/queueingbundle-sqs
• kaliop/queueingbundle-stomp
A design focusing on safety
• The consumer has been split in 2
• The listener is long lived; it spins up a php worker
proc...
Does it scale ?
• The listener waits for the worker to finish before staring another
• Solution: run multiple listeners in...
Multiple protocol support
Kaliop Queueing Bundles
fusesource/stomp-phpaws/aws-sdk-php oldsound/rabbitmq
ActiveMQ ApolloSQS...
Easy to use
• Message encoding taken care of (JSON by default)
• Has events your code can listen to alter the consumption
...
Show me da code
old_sound_rabbit_mq:
connections:
default:
...
producers:
my_producer:
connection: default
exchange_option...
Show me da code
Worker
use KaliopQueueingBundleServiceMessageConsumer;
class MyConsumer extends MessageConsumer
{
/**
* @p...
Show me da code
Lots of console commands available
php app/console kaliop_queueing:queuemessage <queuename> <msg body>
php...
Known limitations
• Assumes good security on the network
• Low throughput
• Spinning up a new php worker process takes tim...
Further evolution
• Instead of spinning off a console command for each message
received, send an http call to a php-fpm pr...
It’s all about eXperience
Thanks for
listening
The end
Prochain SlideShare
Chargement dans…5
×

Rabbits, indians and... Symfony meets queueing brokers

2 453 vues

Publié le

Queuing systems can be integrated in php web apps for a myriad of reasons, but which one is the best choice, and how to easily integrate it?

Publié dans : Internet
  • Soyez le premier à commenter

Rabbits, indians and... Symfony meets queueing brokers

  1. 1. It’s all about eXperience Gaetano Giunta Gilles Guirand Forum PHP Paris 2015/11/24 Rabbits, indians and... Symfony meets queueing brokers
  2. 2. Brokers, don’t they work in Wall Street?
  3. 3. Brokers, don’t they work in Wall Street? Message brokers are a building block of Message oriented middleware. [wikipedia]
  4. 4. Brokers, don’t they work in Wall Street? Message brokers are a building block of Message oriented middleware. [wikipedia] Message oriented middleware (MOM) is software or hardware infrastructure supporting the sending and receiving of messages between distributed systems [same as above]
  5. 5. • RabbitMQ • ActiveMQ • Apollo • Kafka • Kestrel • Amazon SQS • ZeroMQ* • And many more… Let’s try again: can you name any broker?
  6. 6. What do they do?
  7. 7. Where do they differ? • Protocol support (amqp, stomp, …) • Libraries exist for writing clients using standard protocols • Other offer their own SDK • Routing • Amqp has exchanges and queues, Stomp only queues • Wildcards in queue names are supported by most systems • Delivery guarantees • Possible double delivery? • Out of order dispatch
  8. 8. Where do they differ? • Surviving failures • Are messages sent before the client is started buffered or lost? • What happens when a client crashes? • Support for ACK/NACK mechanisms • Clustering • And if the server crashes? • Performances • Batch send/receive – prefetch • Polling vs. persistent connections
  9. 9. Everything clear so far?
  10. 10. Everything clear so far?
  11. 11. A case study: generating MSOffice docs The needs • Produce content in Microsoft Office formats • Many formats for each document • Both Word and Excel docs • XML content generated by CMS • LibreOffice used to generate MS Office and PDF versions • Has anyone tried generating OOXML by hand?
  12. 12. A case study: generating MSOffice docs Reality bites • Slooow • Rest assured that some of the docs will be pretty big • Unreliable • Depending on version, LO crashes from a-bit to almost-always • Race-prone • Two conversion jobs in parallel step on each other
  13. 13. A case study: generating MSOffice docs …and the results are: • Ugly code: lots of polling, copying of files around, manual locking • Does not scale at all: only one conversion process active at any given time Web server PHP Libre Office Waiting processes
  14. 14. TextBook scenario for introducing queues Web server Consumer Rabbit MQ Libre Office PHP
  15. 15. TextBook scenario for introducing queues Web server Consumer ? Rabbit MQ Libre Office PHP
  16. 16. TextBook scenario for introducing queues Web server PHP !!! Rabbit MQ Libre Office PHP
  17. 17. TextBook scenario for introducing queues Web server PHP ! Rabbit MQ Libre Office PHP
  18. 18. TextBook scenario for introducing queues Web server Symfony !!! Rabbit MQ Libre Office Symfony
  19. 19. TextBook scenario for introducing queues Web server Symfony !!! Rabbit MQ Libre Office Symfony
  20. 20. The rationale • Same library handling both sides of the connection • Easier to troubleshoot • Sf Console component is lovely • Everything developed in a single repository • No need to learn new languages • Everything which I can rewrite in PHP, I eventually will* * = 3rd whimsical law of Gaetano
  21. 21. There is a bundle for that oldsound/rabbitmq-bundle • specific to RabbitMQ • supports both producers and consumers php app/console rabbitmq:consumer <name> class FooConsumer implements ConsumerInterface { public function execute(AMQPMessage $msg) { $foo = unserialize($msg->body); echo 'foo '.$foo->getName()." successfully downloaded!n"; } }
  22. 22. Long lived php processes ? • Fatal errors • Memory leaks • Database connections might become broken • Is it in your code or in a dependency?
  23. 23. A new hope • kaliop/queueingbundle • kaliop/queueingbundle-sqs • kaliop/queueingbundle-stomp
  24. 24. A design focusing on safety • The consumer has been split in 2 • The listener is long lived; it spins up a php worker process each time it receives a command • The worker process is a Symfony console command • It handles a single message then exits Symfony listener Symfony worker Libre Office Queue
  25. 25. Does it scale ? • The listener waits for the worker to finish before staring another • Solution: run multiple listeners in parallel W Symfony listener Symfony worker Libre Office Queue Symfony listener Symfony worker Libre Office
  26. 26. Multiple protocol support Kaliop Queueing Bundles fusesource/stomp-phpaws/aws-sdk-php oldsound/rabbitmq ActiveMQ ApolloSQS RabbitMQ Application
  27. 27. Easy to use • Message encoding taken care of (JSON by default) • Has events your code can listen to alter the consumption loop Message received Do stuff Consump tion failed Message consumed Exception
  28. 28. Show me da code old_sound_rabbit_mq: connections: default: ... producers: my_producer: connection: default exchange_options: name: my_producer.exchange type: topic consumers: my_consumer: connection: default exchange_options: name: my_producer.exchange type: topic queue_options: name: my_queue callback: my_symphony_service
  29. 29. Show me da code Worker use KaliopQueueingBundleServiceMessageConsumer; class MyConsumer extends MessageConsumer { /** * @param mixed $body */ public function consume($body) { // do stuff ... } }
  30. 30. Show me da code Lots of console commands available php app/console kaliop_queueing:queuemessage <queuename> <msg body> php app/console kaliop_queueing:consumer <queuename> [-i<driver>] php app/console kaliop_queueing:managedriver list php app/console kaliop_queueing:managequeue list-configured [-i<driver>] php app/console kaliop_queueing:managequeue info <queuename> [-i<driver>]
  31. 31. Known limitations • Assumes good security on the network • Low throughput • Spinning up a new php worker process takes time • Specific features of the different protocols are not (yet) supported • The goal remains to have a uniform API
  32. 32. Further evolution • Instead of spinning off a console command for each message received, send an http call to a php-fpm process • Support for RPC-style calls • Others? • Suggestions are welcome Symfony listener Symfony worker Libre Office Queue HTTP Web server: php daemon
  33. 33. It’s all about eXperience Thanks for listening The end

×