1. Background
Processing
Kerry Buckley
IPRUG 3 November 2010

2. Why?
• Sending e-mail, SMS, tweets etc
• Resizing images
• Generating large PDFs
• Updating search indexes
• etc

3. Approaches
• Process/thread based
• Spawn
• Database-backed
• BackgroundDRb, BackgroundJob,
DelayedJob, JobFu, Background-Fu
• Queue-backed
• Sparrow, Starling, Kestrel, RabbitMQ,
ActiveMQ, SQS, Beanstalkd, Resque

5. Delayed::Job

6. Delayed::Job
• Extracted from Shopify
• Daemon runs with Rails environment
• Use collectiveidea’s fork
• 2.0 for Rails 2.x
• 2.1RC (from GitHub) for Rails 3
• ActiveRecord/MongoMapper/DataMapper

7. Setup
Gemfile:
gem "delayed_job",
:git =>
"git://github.com/collectiveidea/delayed_job.git"
Then run:
$ rails g delayed_job
$ rake db:migrate
$ script/delayed_job start

8. Delaying methods
Mailer.send_alerts(event)

9. Delaying methods
Mailer.delay.send_alerts(event)

10. Options
Mailer.delay(:priority => 3,
run_at => 1.hour.from_now).
send_alerts(event)

11. Running workers
Foreground
$ rake jobs:work
$ script/delayed_job run
Daemonised
$ script/delayed_job start
$ RAILS_ENV=production script/delayed_job -n 4 start

12. Custom Jobs
class NewsletterJob < Struct.new(:text, :emails)
def perform
emails.each do |e|
NewsletterMailer.deliver_text_to_email(text, e)
end
end
end
emails = Customers.all.collect(&:email)
Delayed::Job.enqueue NewsletterJob.new(text, emails)

13. Hooks
def enqueue(job)
end
def before(job)
end
def after(job)
end
def success(job)
end
def error(job, exception)
end
def failure
end

14. Resque

15. Resque
• Created by GitHub
• Jobs persisted to Redis as JSON
• Multiple queues
• Parent/child forking architecture
• Web monitoring interface

16. Setup
Redis:
$ brew install redis
Gemfile:
gem "resque"
Rakefile:
require "resque/tasks"
task "resque:setup" => :environment

17. Delaying methods
class Archive
@queue = :file_serve
def self.perform(repo_id, branch = 'master')
repo = Repository.find(repo_id)
repo.create_archive(branch)
end
end
Resque.enqueue(Archive, repository.id, branch)

18. Running queues
$ QUEUE=queue_name rake resque:work
$ QUEUES=file_serve,warm_cache rake resque:work
$ QUEUES=critical,high,low rake resque:work
$ QUEUE=* rake resque:work
$ COUNT=5 QUEUE=* rake resque:workers

19. Monitoring
$ resque-web

20. Which to use?
Delayed::Job Resque
How much of your site is background work?
Less than 50% More than 50%

21. Demo