First given at RubyConf India 2010

1. Message Queueing in Ruby
An Overview
Pradeep Elankumaran
@pradeep24
pradeep@intridea.com

2. About Us
❖ Intridea is a high-end web consulting shop
❖ Based out of Washington DC, but dispersed all over the United States
❖ Our most popular product is Presently, an enterprise microblogging
platform
❖ Grub.it is our latest product - mobile dish reviewing
❖ Various iPhone & Android apps (Car Finder, Grub.it)
❖ Specialize in web & mobile development using Ruby, Rails, Java, Erlang
and beyond

3. Some background...

4. A request is supposed to
return a response really fast.

5. However, there are certain
actions that may take longer
than a few milliseconds

6. A Few Examples
❖ Delivering e-mail
❖ Image processing (resizing, cropping etc.)
❖ Sending out notifications
❖ Geocoding a location
❖ Talking to other web services

7. These actions, if not
extracted out to run
asynchronously, will slow
down your app

8. Bad user experience

9. What is a
Message Queue?

10. A server that you submit
asynchronous ‘jobs’ to
Things to be done later

11. Interfaces between your Rails
app and the ‘workers’ that
perform each ‘job’

12. ‘Workers’ pull jobs from each
queue and process them
ASAP

13. Queues need to be
FAST & RELIABLE

14. Queue 1: Delayed Job
github.com/tobi/delayed_job

15. Extracted out from
Shopify.com

16. Delayed Job is backed by
your database.

17. Serializes Ruby Job classes
(YAML) into a ‘jobs’ table

18. Easy-to-use syntax

19. rake jobs:run

20. Polls the database
to get new jobs :(

21. A Few Notes about DJ
❖ used by Github exhaustively till they moved to Resque
❖ will get you 80% of the way there, has some nice advanced
features (priority, retries etc.)
❖ should be careful not to overload your jobs table
❖ backed up jobs can lead to query slowdown on your site
❖ suggested that you monitor and profile workers before you
launch on production

22. Queue 2: Starling
github.com/starling/starling

23. Extracted out (infamously)
from early versions of
Twitter

24. Gets a bad rep for systems
with high load, but pretty
great for smaller apps

25. Speaks Memcached Protocol

26. Distributed from the
client-side

27. Easy-to-use Syntax

28. Will have to write your own
workers :(

29. Will have to poll the queue
to get new jobs :(

30. A Few Notes about Starling
❖ persists to RAILS_ROOT/tmp/starling
❖ great for smaller apps, very lightweight
❖ again, will get you 80% of the way there
❖ writing robust, reliable workers is left up to you
❖ polling makes it slow compared to other queues

31. Queue 3: ActiveMQ
activemq.apache.org

32. Mature Apache project.

33. Been around for a while, used
by many, many websites &
companies

34. Written in Java - however,
can connect w/ Ruby using
the STOMP protocol

35. Scripts provided to stop/start
queue server daemon
bin/activemq stop|start|restart

36. Use ActiveMessaging for easy
integration or the STOMP
gem for manual integration

37. ActiveMessaging plugin
❖ developed by ThoughtWorks
❖ provides a mini-framework to publish and subscribe to
various ActiveMQ queues and topics
❖ mature, well-tested

38. Publish from any Ruby object

39. Define your own Processors
script/poller run

40. Comes with a basic
monitoring interface

41. Evented!

42. A Few Notes about ActiveMQ
❖ enterprise-grade
❖ can be clustered (edit XML files)
❖ using ActiveMessaging, no need to write your own workers
❖ STOMP gem is great if you do want to write your own
workers

43. Queue 4: RabbitMQ
rabbitmq.com

44. AMQP
Advanced Message Queuing Protocol

45. AMQP can do lots of stuff...
❖ Point-to-point communication
❖ One-to-many broadcasting (including multicast)
❖ Transactional publication and acknowledgement
❖ High-speed transient message flows
❖ Reliable persistent message delivery
❖ File streaming

46. Fast, Reliable Erlang Queue

47. Careful with the libraries....
❖ @tmm1’s amqp gem - runs off EventMachine, doesn’t play
well with Passenger (asynchronous)
❖ warren - the best of the bunch for Rails (synchronous)
❖ bunny - lightweight (synchronous)

48. Evented!

49. A Few Notes about
RabbitMQ
❖ enterprise-grade
❖ can be clustered using Erlang’s built-in awesome clustering
❖ will have to write your own workers unless you use Workling
❖ very, very fast

50. Queue 5: Resque
github.com/defunkt/resque

51. Resque sits on top of Redis

52. Redis
superfast data structures store
like memcached, but smarter
in-memory for the most part
persisted to disk asynchronously
sets, lists & corresponding operations

53. created by @defunkt
of github.com

54. persists jobs to Redis as
JSON objects

55. Easy to define new Jobs
any Ruby class that responds
to ‘perform’ method

56. Queuing jobs
QUEUE=image_queue rake resque:work
QUEUE=* rake resque:work

57. Also comes with a Sinatra
app that lets you monitor
your queues

58. A Few Notes about Resque
❖ has a dependency on Redis
❖ don’t have to write your own workers
❖ very nice worker fork()
❖ not evented :(

59. Pick the right queue for
the job

60. Do your own load-tests
on your own stack

61. The Care & Feeding of
Worker Daemons

62. More often than not, the
queues are not the problem.

63. Writing workers
in Ruby is easy...

64. ... but writing Ruby workers
that are long-running and
reliable is NOT easy
unless you’re using EventMachine or
really know what you’re doing

65. Why so hard to write Ruby workers?
❖ You will wake up to workers with memory leaks
❖ Workers sometimes lock up
❖ Loading the whole Rails stack for each worker
❖ In general, a pain to manage these worker daemons

66. Use EventMachine
Reactor Pattern
Define callbacks on events
pretty solid & fast

67. fork( )
spin up the worker in the new process
close when done
can kill bothersome workers
easier debugging

68. Workling plugin
github.com/purzelrakete/workling

69. Questions?
@pradeep24
pradeep@intridea.com

70. Something new...

71. Llama
❖ A port of Apache Camel to Ruby
❖ based off EventMachine, so it runs ROCK SOLID
❖ early alpha at this point, looking for helpers
❖ github.com/skyfallsin/llama

72. Llama Code Example

73. github.com/skyfallsin/llama