Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

2015 ZendCon - Do you queue

425 vues

Publié le

Presentation given during the 2015 ZendCon conference on queuing.

Publié dans : Technologie
  • Soyez le premier à commenter

2015 ZendCon - Do you queue

  1. 1. DOYOU QUEUE? ZendCon 2015
  2. 2. I AM MIKE WILLBANKS • Father, Husband, Developer • VP of Development at Packet Power • Twitter: @mwillbanks
  3. 3. Task Producer ConsumerMessages Messages Messages WHAT IS A QUEUE? • Pub/Sub • FIFO buffer • Push / Pull • A way to communicate between applications / systems. • A way to decouple components. • A way to offload work.
  4. 4. WHY QUEUE?
  5. 5. THE LONG RUNNING CASE Request Long-process Send ResponseResponse
  6. 6. THE INTEROP CASE Request Conditioning Service ResponseResponse Web Service Call Send Response
  7. 7. NOTYOUR DATABASE Request Process Response Send Response DB
  8. 8. SO WHY QUEUE • User experience • System Security • Load Distribution • System Reliability
  9. 9. IN PRACTICE You’ve seen this before…
  10. 10. WHATYOU MIGHT QUEUE
  11. 11. COMMUNICATIONS • Emails • SMS • Push Notifications
  12. 12. IMAGES • Conversions • Resize • Thumbnail • Watermark
  13. 13. VIDEO • Conversion • Resampling • Audio overlay
  14. 14. IOT • Receive messages from devices and process responses
  15. 15. PATTERNS
  16. 16. POINTTO POINT Point to Point Channel Receiver Sender
  17. 17. PUBLISH / SUBSCRIBE Publiser Subscribe Channel Subscriber Subscriber Subscriber
  18. 18. MESSAGE BUS Application ApplicationApplication Message Bus
  19. 19. PIPELINE Sender Receiver Point to Point Channel Receiver Point to Point Channel
  20. 20. INVALID MESSAGE Channel Receiver Sender X Invalid Message Channel
  21. 21. PROTOCOLS Or implementations for that matter.
  22. 22. AMQP • AMQP Working Group (Community andVendor) • Platform agnostic protocol. • Completely open, interoperable and broadly applicable. • Many severs available and many client libraries.
  23. 23. STOMP • Simple protocol • Behaviors follow very simple commands. • Most message queues can communicate over STOMP.
  24. 24. Connect Send Disconnect /queue/ msg P H P S T O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack
  25. 25. SQS • Fairly simple protocol • Supports delays, timers, and multiple policies.
  26. 26. SPECIAL PURPOSE Many queue implementations exist that don’t necessarily sit under standards…
  27. 27. XMPP • Best for real-time data. • Leveraging pub/sub can turn it into more of a generic message system. • Multiple libraries available.
  28. 28. SOCKET.IO • New comer • Real-time bidirectional event-based communication • Largely leverages pub/sub
  29. 29. ZEROMQ • The ultimate in message queue flexibility. • Socket library that acts as a concurrency framework.
  30. 30. GEARMAN • Application framework for farming out work. • Job sever for asynchronous or synchronous messages.
  31. 31. BEANSTALKD • Asynchronous Job Queue • Supports delays • Many PHP clients exist
  32. 32. CONSIDERATIONS How do we evaluate our options…
  33. 33. PULLVS. PUSH • Always PULL, whenever possible. • Push eliminates several benefits.
  34. 34. DURABILITY • Memory residence • Persistence • Restart survival
  35. 35. SECURITY • Authentication • Queue permissions / restrictions
  36. 36. DELIVERY • Is the delivery guaranteed? • If a message cannot be delivered how it it handled?
  37. 37. ROUTING • Multiple routing scenarios • Fanout • Direct • Topic • Broadcast
  38. 38. BATCHING • Do it later but in bulk (credit card processing) • Can be done via scheduling (cron)
  39. 39. RECEIPT • Do you get an acknowledgement of receipt?
  40. 40. IMPLEMENTING QUEUES
  41. 41. STARTING POINTS
  42. 42. PUSHING MESSAGES <?php class UserService { public function save($user) { $this->db->save($user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', [ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); } }
  43. 43. HANDLING MESSAGES <?php $stomp = new Stomp('tcp://localhost:61613'); $stomp->subscribe('/queue/email'); while (true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body); mail($email->to, $email->subject, $email->message, $email->headers); }
  44. 44. MESSAGES
  45. 45. MESSAGE CONSIDERATIONS • Message Format • Message Contents
  46. 46. SERIALIZE O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s: 3:"baz";}}}}
  47. 47. WORKERS
  48. 48. WORKER CONSIDERATIONS • Should do ONE thing and ONE thing well. • Should attempt to be as quick as possible in handling that type. • Should be able to be scaled horizontally.
  49. 49. HANDLING WORKERS • Prevent Memory Leaks • memory_get_usage • Handle Signals! • pcntl_signal
  50. 50. ABSTRACTIONS We want to make this easy…
  51. 51. <?php interface QueueInterface { public function __construct(Stomp $stomp, $queue); public function dispatch(); public function publish(array $message); public function work(StompFrame $message); }
  52. 52. <?php class AbstractQueue implements QueueInterface { protected $stomp; protected $queue; protected $signal; public function __construct(Stomp $stomp, $queue) { $this->stomp = $stomp; $this->queue = $queue; } protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); } if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } } protected function signal($signal) { $this->signal = $signal; }
  53. 53. public function dispatch() { $this->prepare(); while (true) { if ($this->signal) { break ; } if (!$this->stomp->hasFrame()) { $this->wait(); continue ; } $frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); } $this->stomp->ack($frame); } } protected function wait() { sleep(1); } protected function validate(StompFrame $message) { return false; } public function publish(array $message) { return $this->stomp->send($this->queue, json_encode($message)); }
  54. 54. <?php class EmailQueue extends AbstractQueue { public function validate(StompFrame $message) { if (!array_key_exists('to', $message)) { return false; } return true; } public function work(StompFrame $message) { $mail = json_decode($message); mail($mail->to, $mail->subject, $mail->message); } }
  55. 55. BOOTSTRAPPING Leverage your existing infrastructure as much as possible!
  56. 56. <?php declare(ticks=1); include 'vendor/autoload.php'; $app = ZendMvcApplication::init(include 'config/application.config.php'); $sm = $app->getServiceManager(); if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker <name>nn"); exit(1); } $name = $argv[1]; try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (Exception $e) { fprintf(STDERR, "%sn", $msg); exit(1); } $consumer = null; echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);
  57. 57. EVENTS
  58. 58. SERVICESTRIGGER EVENTS <?php use ZendEventManagerEventManagerAwareTrait; class UserService { use EventManagerAwareTrait; public function save($user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }
  59. 59. ATTACH EVENTS <?php use ZendServiceManagerServiceManager; $sm = new ServiceManager(); $service = $sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish([ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });
  60. 60. HANDLING PROGRESS • Keep track by using a generic handler • Or, keep track via your database.
  61. 61. TOOLING
  62. 62. SUPERVISOR • Daemon that runs on the server. • Monitors programs and keeps them running in case of failure. • Handles logging.
  63. 63. PAINLESS INSTALLATION sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo service supervisor start
  64. 64. EXAMPLE PROGRAM CONFIGURATION [program:emailworker] command=/usr/bin/php /var/www/worker "MyProjectQueueEmail" process_name=%(program_name)s_%(process_num)d numprocs=2 numprocs_start=2 user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15
  65. 65. SUPERVISORD MULTI SERVER MONITORING TOOL https://github.com/mlazarov/supervisord-monitor
  66. 66. WHEN BADTHINGS HAPPEN
  67. 67. QUEUE BACKUP • Most of all issues with queues are that a queue has backed up.
  68. 68. WORKER EXCEPTIONS • Broken code causes hardships, but recoverable!
  69. 69. • https://pixabay.com/en/autobahn-accident-germany-car-road-837643/ • https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/ • https://pixabay.com/en/airplanes-line-runway-military-713662/ • https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/ • https://pixabay.com/en/user-top-view-office-keyboard-154199/ • https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/ • https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/ • https://pixabay.com/en/umbrellas-red-blue-patterns-205386/ • https://pixabay.com/en/spot-runs-start-la-stadion-862274/ • https://pixabay.com/en/artistic-the-art-of-abstraction-948588/ • https://pixabay.com/en/boots-work-boots-shoes-647035/ • https://pixabay.com/en/meerkat-watch-guard-cute-676944/ • https://pixabay.com/en/broken-window-hole-glass-damage-960188/ • https://pixabay.com/en/police-security-safety-protection-869216/ • https://pixabay.com/en/parcel-package-packaging-box-575623/ • https://pixabay.com/en/directory-signposts-trail-direction-494457/ • https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/ • https://pixabay.com/en/phone-communication-call-select-735060/ • https://pixabay.com/en/receipt-note-paper-bill-document-575750/ • https://pixabay.com/en/tools-construct-craft-repair-864983/ • https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/ • https://pixabay.com/en/network-iot-internet-of-things-782707/ • https://pixabay.com/en/mobile-phone-smartphone-app-426559/ • https://pixabay.com/en/mr-site-build-crane-baukran-462074/ • https://pixabay.com/en/film-projector-movie-projector-738806/ • https://pixabay.com/en/calves-legs-human-standing-on-540519/ • https://pixabay.com/en/notebook-pages-opened-paper-note-820078/ • https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/ • https://pixabay.com/en/spray-household-surface-shine-315164/ • https://pixabay.com/en/industry-crafts-gears-mechanical-94448/ • https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/ • https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/ • https://pixabay.com/en/temple-china-door-handle-840526/ • https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/ Image Credits THANKYOU! http://joind.in/talk/view/15538

×