2. What the wibblefish?
All about offloading work, and doing things in
‘realtime’
Make applications that complete tasks in a
distributed/scalable fashion.
Doing work outside the request/response cycle
of a UI/API.
3. Work to offload
Noti cations (SMS, Email, Postal mail).
Logging.
Import & export.
Process image & video.
Reporting.
Webhooks.
4. Isn’t cron good enough?
Cron tabs are great with one server.
Cron doesn’t scale to multiple machines.
Cron doesn’t solve the whole problem space.
5. Problems with Cron
Isn’t suitable for ‘real-time’ results.
Requires a set schedule. You might spin up
processes that do nothing. Or you might not
have enough capacity.
Parallelizing is a pain.
6. Queues to the rescue!
Queues accept and dispatch events.
Events can contain almost anything*.
Events can be roughly equal to ‘jobs’.
A single event can trigger lots of downstream
work.
7. Architectural benefits
Separation of concerns.
Collection of mini/small applications.
No monolithic applications.
Decoupled, and easier to test.
Easy to scale. Busy service = more of them.
16. Exchanges
Exchanges abstract queues from the producer.
Route messages to queues, that are bound by
routing keys.
Allows consumers to listen to speci c routing
keys.
Multiple consumers can listen to the same
events.
22. Topic subscription
Consumers can also listen to topics. Which are
keys separated by dots.
e.g page_view.search, page_view.update
Consumer could listen to page_view.*
23. Producing
Can create messages with many tools.
PHP, python, node, ruby, etc.
Messages can contain any stringy data (JSON)
24. Data for messages
Only string-y things can be stored in messages.
JSON encoded data in message.
Key to value stored in Redis/Memcache.
Keep messages small. RabbitMQ performs
better with small messages.
25. Consuming
Use a daemon process.
Daemons in PHP can be done, but they kinda
suck.
Usually use sleep(2) or other workaround.
sleep(2) moves you away from ‘real time’.
43. Limitations of using shells
Daemon and shell processes have to be on the
same box.
This can be limiting as you scale, depends on
how you structure applications.
44. Solutions
Use HTTP.
Have thin daemons dispatch back to the front-
end application using HTTP.
Use Basic/Digest authentication.
Use checksums.