RestMQ
Redis based Message Queue

WHAT !?
RestMQ - Message Queue based on Redis
Data manipulation:
HTTP GET/POST/DELETE, COMET and WebSockets
No signaling or specific protocol (optional JSON protocol)
http://www.restmq.com/
Gleicon Moraes
http://github.com/gleicon
http://zenmachine.wordpress.com
http://www.7co.cc

Key/Value storage
Think memcached, GET and SET operations
[key] = value
>> SET key meh
>> GET key
meh
>> SET fucounter 0
>> INCR fucounter
1
>> GET fucounter
1

Redis
Key-Value database
Atomic operations
Semi-persistent or persistent storage
Publish/Subscription channels
Different Data types: Sets, Sorted Sets, Lists, Hashes
http://code.google.com/p/redis/
http://github.com/antirez/redis
http://rediscookbook.org/

Redis
Data types
Strings
Lists
Sets and Ordered Sets
Hashes
Publish/Subscribe channels

Redis
Things you can do:
Cache
Inverted indexes
Fast counters
Throttle control
Cooler stuff at Redis Cookbook
Load Balancing
Things you can't do:
Search inside all values for a given string

Redis
Things you can't do:
- Search inside all values for a given key
- Search inside all values for a given key
- Search inside all values for a given key

Message Queues
Does everything really needs to be tightly coupled ?
Order: first come, first served
Transactions: there is no transaction, just tracking
Job Scheduling
Publish/Subscribe
Map/Reduce

Message Queues
SMTP: Oldest Message Queue around
Avoid hitting your DB real time with MQ
Real time: Ratings, Voting and Comments
Twitter
Do I need NoSQL or I really need to clean my mess ?

RestMQ - Endpoints Diagram

RestMQ - Messages
/q/<queue> : Queue handling (GET/POST/DELETE)
$ curl -X POST -d "value=foobar" http://localhost:8888/q/testq
testq:1
$ curl -X POST -d "value=foobar" http://localhost:8888/q/testq
testq:2
$ curl http://localhost:8888/q/testq
{
"count": 0,
"value": "foobar",
"key": "testq:1"
}

RestMQ - Messages
/c/<queue> : Comet Endpoint
$ curl http://localhost:8888/c/testq
(hangs until there is a message to be received)
/ws/<queue> : WebSocket Endpoint
Needs the proper javascript code

RestMQ - Messages
JSON Protocol inspired by Amazon SQS
First prototype at http://jsonqueue.appspot.com
add get
{ {
"cmd" : "add", "cmd" : "get",
"queue" : "testq", "queue" : "testq",
"value" : "abacab" }
}
del take
{ {
"cmd" : "del", "cmd" : "take",
"queue" : "testq", "queue" : "testq",
"key" : "testq:31" }
}

RestMQ - Status
/stats/<queue> : Queue status
$ curl http://localhost:8888/stats/test
{
"queue" : "test",
"redis": "127.0.0.1:6379 - 10 connection(s)",
"len": {"len": 1}
}
/stats : Server status
$ curl http://localhost:8888/stats/
{
"count": 4,
"queues": [ "devops:queue", "test3:queue", "test:queue", "test2:queue"],
"redis": "127.0.0.1:6379 - 10 connection(s)"
}

RestMQ - Controls
/p/<queue> : Distribution Policy for COMET and WS
Round Robin a message for each consumer
Broadcast a message to all consumers
/control/<queue> : Queue start/stop
Pause all COMET consumers
(useful for job schedulers)

RestMQ - Data structure Diagram

RestMQ - Algorithm
Algorithm for pushing messages into a queue named 'testq'
SADD testq into a SET called queueset
INCR a per-queue UUID generator called testq:uuid
SET testq:<<id>>, message
LPUSH testq:<<id>> in to a LIST named testq:queue
Works for new and existing queues. On-the-fly queue creation.

RestMQ - Algorithm
Algorithm for reading messages from a queue named 'testq'
RPOP a key from a LIST named testq:queue
GET the value associated to this key
For non-destructive GET operations:
LINDEX -1 to get a key in testq:queue LIST
INCR key:refcount +1
GET the value stored for key

RestMQ - Implementation
(the core functionality can be implemented on any language)
Main branch 'Enterprise Edition'
Python
Twisted
Cyclone
Redis async client
Embedded version
Sinatra + Ruby
Reduced endpoints (GET/POST)

RestMQ - Sinatra Implementation
http://gist.github.com/524240
GET /q - List all queues

RestMQ - Sinatra Implementation
http://gist.github.com/524240
GET /q/<queue> - Get a message from <queue>

RestMQ - Sinatra Implementation
http://gist.github.com/524240
POST /q/<queue> - Insert a message in <queue>

RestMQ - Map/Reduce

Questions ?

Thanks