2. Who am I?
➔ Undergraduate student.
➔ Involved with:
✔ Mozilla as a Contributor and Rep
✔ Drupal as a Contributor
➔ Twitter: @vaidikkapoor
➔ Github: vaidikkp
➔ Web: vaidikkapoor.info
8. What can Queues help with?
➔ Background Processing
✔ Data Processing
✔ Media Processing
✔ Updating Caches
➔ Anything that you want to offload off your
server.
➔ Improve the overall User Experience
10. Redis + HotQueue
PRODUCER:
from hotqueue import HotQueue
queue = HotQueue("myqueue", host="localhost", port=6379, db=0)
queue.put(message)
CONSUMER / WORKER:
from hotqueue import HotQueue
queue = HotQueue("myqueue", host="localhost", port=6379, db=0)
while True:
message = queue.get()
# do something awesome
11. Redis + PyRes
➔ Python clone of Github's Resque
➔ Offers a lot more features than HotQueue
✔ Job failure handling
✔ Queue status information
➔ Comes with Monitoring System written in Ruby
12. Redis + PyRes
PRODUCER:
from pyres import ResQ
r = Resq()
class WebPage:
queue = “screenies”
@staticmethod
def perform(urll):
# save the screenshot
r.enqueue(WebPage, 'http://python.org')
CONSUMER:
./pyres_worker screenies
17. Criteria for Broker Selection
➔ Difficulty in Recovery
➔ Relatively low level of required maintenance
➔ Ease of Deployment
➔ Durability
➔ Persistence
➔ Community Support
➔ Cluster Support
➔ What language is it written in?
20. Celery
➔ An amazing Task Manager
➔ Batteries Included
➔ Uses RabbitMQ as broker (or almost anything)
➔ Libraries for most of the common web frameworks
like Django, Flask, Pyramid
➔ Supports multiprocessing.
➔ Distribute tasks easily.
➔ Makes life easy
35. Review
➔ Choose a system according to needs
➔ Build a robust system
➔ Integrate seamlessly with your UI
➔ Isolate & Recycle: make it a habit
➔ Improve the overall UX!
➔ Please everyone, even yourself!