Using Redis as a Message Queue

1. Redis
As a Message Queue
1Wednesday, October 9, 13

2. The Plug
Essentially, a screenshot site
MMORPG (Massively-Multiplayer, Online
Role-Playing Game) niche market
Site visitors browse games, games have
albums
Players can upload screenshots
2Wednesday, October 9, 13

3. The Architecture
(2) 6-core CPU, 16GB RAM, 6-drive 3TB RAID10
(1) 8-core CPU, 16GB RAM, 2-drive 64GB RAID1 SSD, 2-drive 320GB RAID1
(2) layer-4 ipvsadm load balancers in master/backup failover via
keepalived
(3) PostgreSQL master/slave database
(3) nginx+php-fpm (5.5) app servers
Replicated file system via GlusterFS for images
Memcached instance for php sessions
Redis for caching and message queue
Amazon Cloudfront for CDN (content delivery network)
PhalconPHP framework, Gmagick, Twitter Bootstrap v3.0, REST API
3Wednesday, October 9, 13

4. Message Queue?
Push messages into a stack
Pop (pull) messages off a stack
You put (push) data (messages) into the
queue (stack)
Messages are things you want to process
later by pulling them from the queue
4Wednesday, October 9, 13

5. Messages?
A “message” is just some data
Could be an array
Could be a string
5Wednesday, October 9, 13

6. The Use-Case
When a user uploads images, it is bad to
block the user while your server processes
the photos
Image resizing
Creating multiple versions
Creating database entries
6Wednesday, October 9, 13

7. The User Flow
User selects images
Images are uploaded to app server
App server generates a UUID filename and
copies the images to the replicated file system
App server increments a Redis counter
App server creates an array with uuid, user id,
game id and pushes this “message” into the Redis
list (message queue)
7Wednesday, October 9, 13

8. Batch Processing
A separate PHP/other process running on any of
the app servers (crontab, forking daemon) pops
the Redis list to get a message
By using GlusterFS as a replicated file system all
app servers have all the uploads/images
When a message is obtained from Redis queue, it
processes the file upload, creates various image
sizes
Creates a database entry in PostgreSQL
8Wednesday, October 9, 13

9. Lets push!
// Create a new redis connection
$r = new Redis();
$r->connect($host, $port);
// Create the message array
$message = [‘uuid’ => $uuid, ‘userId’ => $userId, ‘gameId’ => $gameId];
// Get a new message id by implementing the builtin increment feature of Redis
$id = $r->incr('mmoscreens:screens-queue');
// Push the id into the queue (Redis list)
$r->lpush('mmoscreens:screens-queue', $id);
// Set the $message into Redis
$r->hmset("mmoscreens:screens-queue:$id", $message);
9Wednesday, October 9, 13

10. Lets pull!
// Create a new redis connection
$r = new Redis()->connect($host, $port);
// pop the message ID from the queue, or block until one is available
$id = $r->brPop('mmoscreens:screens-queue', BLOCK_WAIT_TIME);
// brPop returns [listName, element]
isset($id[1]) && $id = $id[1];
// Get the message from Redis
$message = $r->hgetall("mmoscreens:screens-queue:$id");
$message:
[
‘uuid’ => ‘fef4430c-e4a8-4874-b380-63edd2fa8472’,
‘userId’ => 10611,
‘gameId’ => 8
]
10Wednesday, October 9, 13