What is Redis and what can you really do with it? Data structures Command processing Persistence Recipes Redis roadmap

1. Redis for serious Social Media
Performance
Chris Mordue @cmordue
March 22, 2012

2. Outline
What is Redis and what can you really do with it?
●
Data structures
●
Command processing
●
Persistence
●
Recipes
●
Redis roadmap
●

3. What is Redis
Many things to many people:
Cache (Memcache replacement)
●
Queueing or Pub/Sub system
●
Data structure server
●
Advanced Key/Value Store
●
What isn't it?
Solution to everything
●
Easily scalable across
●
Object/Graph/Relational DB
●
although many structures can be modeled
○
Author
Salvatore - sponsored by VMware
●

4. Use cases
● Memcached replacement Autocomplete
●
● Duplicate detector Twitter
●
● FIFO/LIFO Queue digg / Hacker News
●
● Priority Queue Social activity feed
●
● Distributed hashmap
○ e.g. url shortener
● UID generator
● Game high scores
● Geolocation lookup
○ lat & long -> city
● Real-time analytics
● Metrics DB
● API throttling (rate-limits)

5. Data structures
Types:
Strings (regular or bitmap)
●
get/set (& not exist), setbit/getbit, incr[by]/decr[by], append
○
Lists
●
push/pop, set, trim, len, range, blocking/non-blocking
○
Hashsets
●
get/set, incrby, exists, del, keys, vals, getall
○
Sets
●
add/rem, pop, diff/union/inter (& store), move
○
● SortedSets
○ add/rem, rank, range/revrange, interstore/unionstore

6. Command processing
Execution Basics
● IO of requests/responses on other thread(s)
● Commands are processed in serial in a single thread
○ If you want to use more CPUs, shard your data client-side
● Superfast processing of commands
Command groups
● Request/response
● Pipelined: Several requests / responses
● Transactions:
○ Traditional: Several requests processed in order atomically
○ Check-And-Set: Request(s), data manipulation in client, more
requests
● Scripts (v2.6): execute client-defined Lua scripts in the server for
more complex actions

7. Persistence
Periodic Snapshots
Append Only File

8. Recipes - UID Generator
Objective: Create unique ids
Implementation*:
● incr global:nextDocId
● The response is a unique id
Performance: >100.000/s
* from http://redis.io/commands/incr

9. Recipes - Duplicate Detector
Objective: Determine if an object has already been processed
Implementation:
● sadd <objectId>
● If the response is 1, it's new. If 0, it already existed
Gotchas:
● use lru eviction rules to remove old data to keep the data size contained
Examples:
● Socialmetrix's twitter-engine

10. Recipes - Memcached
Objective: Application cache
Implementation:
serialize your objects as a string
●
get/set
●
expire (time to live)
●
Performance >= memcache
Gotchas:
Memcache scales horizontally out of the box, redis does not
●
Examples:
Integration with Play! Framework's Cache:
●
http://www.playframework.org/modules/redis
○
https://github.com/tkral/play-redis
○
Garantia Data: providing hosted redis/memcached instances
●
http://www.garantiadata.com/
○
Image from http://www.garantiadata.com/

11. Recipes - API Rate Limiter
Objective: Recognize when more than E events happen in time T seconds by
user U.
Implementation 1*: Implementation 2:
FUNCTION LIMIT_API_CALL(ip) FUNCTION LIMIT_API_CALL(ip)
ts = CURRENT_UNIX_TIME() ts = CURRENT_UNIX_TIME()
keyname = ip+":"+ts keyname = ip+":"+ts
current = GET(keyname) MULTI
current = INCR(keyname, 1)
IF current != NULL AND current > 10 THEN
EXPIRE(keyname,10)
ERROR "too many requests per second" EXEC
ELSE IF current < 10 THEN
MULTI PERFORM_API_CALL()
INCR(keyname,1) ELSE
EXPIRE(keyname,10) ERROR "too many requests per second"
END
EXEC
PERFORM_API_CALL()
END
Gotchas:
● make sure you set expires on the keys in a transaction because set or incr
removes old expires values
Examples:
* from:http://redis.io/commands/incr

12. Recipes - Twitter clone
Objective: Duplicate the functionality of twitter
Implementation:
● Generate Ids with String: INCR global:nextUserId => 1000
Username to id with String: SET username:antirez:uid 1000
● Followers/Follwings with 2 sets: uid:1000:followers
● Timeline with List: uid:1000:posts
● Tweeting:
○ Create tweet id with String: INCR global:nextPostId => 10343
○ Store tweet: SET post:10343 "$owner_id|$time|Having fun w/ Retwis"
○ Get Followers list and push tweetId to allow followers timelines:
■ LPUSH uid:$followeruid:posts 10343
● View posts:
○ Get list of postIds: LRANGE uid:1000:posts 0 50
○ For each postId, get posts to display: GET post:$postId
From:
● http://redis.io/topics/twitter-clone

13. Recipes - Twitter clone cont.
Code:
● https://github.com/SpringSource/spring-data-keyvalue-examples
● http://retwis.antirez.com/
● http://code.google.com/p/redis/downloads/list
Performance:
"On a very slow and loaded server, apache benchmark with 100 parallel clients
issuing 100000 requests measured the average pageview to take 5
milliseconds. This means you can serve millions of users every day with just a
single Linux box, and this one was monkey asses slow!" - Salvatore

14. Redis roadmap
v2.6:
Server side scripting with Lua (v5.1): http://www.lua.org/manual/5.1/
●
Lots of possibilities with scripts - invent your own
commands
v3.0:
● Cluster support
● In dev for many months

15. Other Recipes
Old cookbook (soon to be updated): http://rediscookbook.org/
GeoHashing: https://github.com/doat/geodis
Autocomplete: http://patshaughnessy.net/2011/11/29/two-ways-of-using-redis-
to-build-a-nosql-autocomplete-search-index
Slides 17-20: http://www.slideshare.net/dvirsky/introduction-to-redis-version-2
Digg clone design: http://groups.google.com/group/redis-
db/browse_thread/thread/94d0dd1a24bf400e/98a0f5935380a075?
lnk=gst&q=digg#98a0f5935380a075
Lots of other recipes: http://groups.google.com/group/redis-db
Sinatra dashboard for redis: https://github.com/steelThread/redmon

16. ¡Gracias!
Chris Mordue
Software Architect
@cmordue