Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Faster PHP apps using Queues and Workers

4 100 vues

Publié le

PHP apps typically perform tasks in a synchronous manner; Resizing an image or sending a push notification. For most applications this works well, but as apps grow or experience increased traffic, each task adds extra milliseconds to a request, leaving users waiting.

A common solution is to defer these tasks to the background using a cron task. However, there is a better way. Job queues not only help to decouple your application and improve resilience but will also cut request times.

In this talk I we’ll explore some common queue systems; the features and tradeoffs of each solution, what to queue, refactoring existing code into jobs, and running workers. By the end you’ll be ready to build your next app one job at a time.

Publié dans : Internet
  • Soyez le premier à commenter

Faster PHP apps using Queues and Workers

  1. 1. QUEUES & WORKERS FASTER PHP APPS USING PHP DORSET 6/6/16
  2. 2. RICHARD BAKER twitter.com/r_bake_r github.com/rjbaker uk.linkedin.com/in/richjbaker
  3. 3. We’re hiring!
  4. 4. BACKGROUND MOBILE API ▸ 140+ apps ▸ ~4.5m users ▸ REST/JSON ▸ Apache + PHP + ElasticSearch ▸ 20+ servers / various tasks ▸ Hosted on AWS
  5. 5. THE PROBLEM TRAFFIC
  6. 6. THE PROBLEM LATENCY GROWS
  7. 7. ELIMINATE EXPENSIVE, TIME CONSUMING OPERATIONS AND CALLS TO EXTERNAL SERVICES
  8. 8. USE MYSQL
  9. 9. WHAT TO QUEUE MYSQL AS A QUEUE job_id job_name data status 1 sendEmail {payload} completed 2 sendPush {payload} processing 3 requestThing {payload} waiting 4 resizeSelfie {payload} waiting
  10. 10. WHAT TO QUEUE WHY NOT USE TRANSACTIONS TO OBTAIN A LOCK?
  11. 11. THE DATABASE IS NOT A QUEUE. THE DATABASE IS NOT A QUEUE. Stephen Corona WHAT TO QUEUE https://www.scalingphpbook.com
  12. 12. USE A QUEUE!
  13. 13. QUEUES & WORKERS MESSAGE QUEUES VS JOB QUEUES ▸ Kind of similar ▸ Most job queues built upon some kind of message queue ▸ Broker messages between systems ▸ Provide transport, storage and protocol ▸ Job queues abstract the lower level message component ▸ Integrate with most applications fairly easily
  14. 14. QUEUES & WORKERS JOBS VS SCHEDULED TASKS ▸ Run at a predefined point in time ⏰ ▸ May repeat at regular intervals or according to calendar 📅 ▸ Typically triggered by cron or other scheduler 🔫 ▸ Scheduled tasks can trigger jobs! 💥
  15. 15. CHOOSE A QUEUE WITH FEATURES BEST SUITED TO YOUR APPLICATION
  16. 16. CHOOSING A JOB QUEUE CONSIDERATIONS ▸ Job priority & time sensitivity ▸ Job ordering and consistency (FIFO) ▸ Payload size limits ▸ Message data type & protocol ▸ Support for other languages / client libraries ▸ Failure management / retry policy ▸ Fault tolerance & redundancy ▸ One-time delivery guarantee ▸ Monitoring & statistics ▸ Distribution by task to specific workers e.g. Video encoding
  17. 17. CHOOSING A JOB QUEUE BEANSTALKD ▸ Protocol similar to Memcached ▸ Clients need to know about all Beanstalkd servers (like memcached!) ▸ Beanstalkd servers can persist jobs, handle restarts without losing jobs ▸ Uses “tubes” to differentiate different queues ▸ Supports job TTR. Failed/hung jobs get put back into queue. ▸ Supports blocking. Client connects and waits for a new job. ▸ Requires setup and maintenance ▸ Loads of client libraries http://kr.github.io/beanstalkd/
  18. 18. CHOOSING A JOB QUEUE AMAZON SQS ▸ SAAS - Already using AWS, literally no setup ▸ Massively redundant, cheap, maintenance free ▸ HTTP/JSON under the hood, simple ▸ Supports long-polling. ▸ Best effort FIFO (no guarantees) ▸ No concept of job priority. Use different queues. ▸ Retry policy allows jobs to reappear in queue if not completed in specified time ▸ Configurable number of retries ▸ Queue stats and alarms integrate with autoscaling ▸ Scale worker instances based on queue length/backlog/rate https://aws.amazon.com/sqs/
  19. 19. CHOOSING A JOB QUEUE OTHER POPULAR QUEUES ▸ Celery (backed by RabbitMQ) - http://www.celeryproject.org ▸ php-resque (backed by Redis) -https://github.com/ chrisboulton/php-resque ▸ Kafka - http://kafka.apache.org ▸ Gearman - http://gearman.org ▸ Iron.io (SAAS) - https://www.iron.io ▸ Loads more - www.queues.io
  20. 20. PROCESSING JOBS
  21. 21. PROCESSING JOBS WORKER PROCESS ▸ Essentially an infinite loop ▸ Executed on command line ▸ Asks queue for new job ▸ Resolves job method ▸ Execute with payload ▸ Delete job from queue ▸ Repeat
  22. 22. PROCESSING JOBS <?php $queue = new Queue(); while(true) { $job = $queue->pop('queue-name'); try { if ($job->execute()) { $job->delete(); } else { $job->release(); } } catch (Exception $e) { $job->release(); } }
  23. 23. IMPROVING THE WORKER
  24. 24. pcntl_signal_dispatch();
  25. 25. PROCESSING JOBS PROCESS CONTROL EXTENSIONS (PCNTL) ▸ Respond to unix process signals ▸ Gracefully stop worker processes ▸ Complete current job before exiting ▸ Careful if using Apache mod_php on same server ▸ http://php.net/manual/en/book.pcntl.php
  26. 26. IMPROVED WORKER <?php namespace Demo; use DemoQueueQueueInterface; class Worker { protected $shouldRun = true; protected $queue; public function __construct(QueueInterface $queue) { declare(ticks = 1); $this->queue = $queue; pcntl_signal(SIGTERM, [$this, 'signalHandler']); pcntl_signal(SIGINT, [$this, 'signalHandler']); pcntl_signal(SIGQUIT, [$this, 'signalHandler']); }
  27. 27. public function run($queueName) { echo "Starting worker on queue '{$queueName}' n"; while ($this->shouldRun) { $job = $this->queue->pop($queueName); try { if ($job->execute()) { $job->delete(); } else { $job->release(); } } catch (Exception $e) { $job->release(); error_log($e->getTraceAsString()); } pcntl_signal_dispatch(); } } public function signalHandler($signal) { switch ($signal) { case SIGTERM: case SIGINT: case SIGQUIT: echo "Job completed. Exiting... n"; $this->shouldRun = false; break; } } }
  28. 28. IMPROVED WORKER WTF IS DECLARE(TICKS = 1);? ▸ Officially deprecated ▸ Triggered after php has executed a certain number of statements ▸ Interacts with pcntl_signal_dispatch() ▸ I admit i’ve not fully tested this with PHP7.0
  29. 29. PROCESSING JOBS KEEPING WORKERS RUNNING
  30. 30. PROCESSING JOBS SUPERVISOR ▸ Process manager in similar vein to forever, pm2, php-fpm ▸ Runs as service ▸ Starts and restarts php worker processes ▸ Has CLI client (supervisorctl) ▸ Web interface ▸ Easy to install and configure http://supervisord.org
  31. 31. PROCESSING JOBS SUPERVISOR CONFIG [program:alertworker] command = /usr/bin/php /path/to/queueRunner.php -q=prod-alerts autorestart = true autostart = true directory = /path/to/scripts environment = DEPLOYMENT='production' exitcodes = 0,2 numprocs = 1 numprocs_start = 0 priority = 999 startretries = 3 startsecs = 4 stderr_capture_maxbytes = 1MB stderr_events_enabled = false stderr_logfile = AUTO stderr_logfile_backups = 10 stderr_logfile_maxbytes = 50MB stderr_syslog = false stdout_capture_maxbytes = 1MB stdout_events_enabled = true stdout_logfile = AUTO stdout_logfile_backups = 10 stdout_logfile_maxbytes = 40MB stdout_syslog = false stopsignal = TERM stopwaitsecs = 10 umask = 022 user = worker
  32. 32. DEMO TIME
  33. 33. THANKS FOR LISTENING! twitter.com/r_bake_r github.com/rjbaker uk.linkedin.com/in/richjbaker

×