Presentation covers basics of Redis: data structures, data modeling examples, transactions, pipelining, best practices and different recommendations

1. Redis Basics
by Arthur Shvetsov
CoreValue/MalkosUA

2. Agenda
● What is Redis?
● Data structures overview
● Data modeling examples
● Keys expiration
● Redis transactions
● Redis pub/sub
● Pipelining
2

3. What is Redis?
● In-memory key-value store, with persistence
● Open source, written in C
● Extremely fast and lightweight
● Binary-safe keys with supported expiration
● High level data structures
● Support for atomic operations and transactions
● Internal scripting with LUA
● Master-slave replication
● Tons of client libraries for all major languages
3

4. History
● REmote DIctionary Server
● Released in March 2009 by Salvatore
Sanfilippo
● Built in order to scale http://lloogg.com/
● According to the monthly ranking by DB-
Engines.com, Redis is the most popular key-
value store
4

5. How to install?
● http://redis.io/download
● The Redis project does not officially support
Windows.
● Microsoft Open Tech group develops and
maintains this Windows port targeting
Win64.
5

6. Redis implementation details
● Redis is a TCP server using the client-server model.
● Redis protocol - simple and efficient text-based
protocol called RESP (REdis Serialization Protocol).
● Every Redis command is atomic, including the ones
that do multiple things.
● Redis is single-threaded, which is how every
command is guaranteed to be atomic. While one
command is executing, no other command will run.
6

7. Data structures overview
Key
● STRING - binary-safe string with max allowed size 512 MB
Value
● Primitives
○ STRING - Strings, integers, or floating point values
● Containers (of strings)
○ LIST - Linked list of strings
○ SET - Unordered collection of unique strings
○ HASH - Unordered hash table of keys to values
○ ZSET - Ordered mapping of string members to floating-
point scores, ordered by score
7

8. Redis STRING
The Redis String type is the simplest type
of value you can associate with a Redis
key. Integer and float numbers can be
stored as STRING type in Redis.
hello
world
string
Key name Type of value
Value stored
Command What it does
GET Fetches the data stored at the given key
SET Sets the value stored at the given key
DEL Deletes the value stored at the given key (works
for all types)
8

9. Redis STRING examples
redis> SET hello world
OK
redis> GET hello
"world"
redis> DEL hello
(integer) 1
redis> GET hello
(nil)
redis> SET counter 100
OK
redis> INCR counter
(integer) 101
redis> INCR counter
(integer) 102
redis> INCRBY counter 50
(integer) 152
9

10. Redis STRING use cases
● Use Strings to store serialized JSON/XML/etc data
● Use Strings as atomic counters using commands in
the INCR family: INCR, DECR, INCRBY.
● Append to strings with the APPEND command. Use
Strings as a random access vectors with
GETRANGE and SETRANGE.
10

11. Redis LIST
LISTs in Redis is a sequence of order
strings, implemented using a Linked
List.
list-key
item
item2
item
list
Key name Type of value
List of values,
duplicates possible
Command What it does
RPUSH Pushes the value onto the right end of the list
LRANGE Fetches a range of values from the list
LINDEX Fetches an item at a given position in the list
LPOP Pops the value from the left end of the list and
returns it
11

12. Redis LIST implementation details
● The max length of a list is 232
- 1 elements
(4294967295, more than 4 billion of elements per list).
● Redis lists are implemented via Linked Lists.
● The operation of adding a new element in the head or
in the tail of the list is performed in constant time.
● Accessing an element by index is very fast near the
head and tail of the list, but O(N) operation in the
middle.
12

13. Redis LIST examples
redis> RPUSH list-key item
(integer) 1
redis> RPUSH list-key item2
(integer) 2
redis> RPUSH list-key item
(integer) 3
redis> LRANGE list-key 0 -1
1) "item"
2) "item2"
3) "item"
redis> LINDEX list-key 1
"item2"
redis> LPOP list-key
"item"
redis> LRANGE list-key 0 -1
1) "item2"
2) "item"
13

14. Redis LIST use cases
● Model a timeline, for example in a social network, using
LPUSH in order to add new elements in the user time
line, and using LRANGE in order to retrieve a few of
recently inserted items.
● You can use LPUSH together with LTRIM to create a
list that never exceeds a given number of elements, but
just remembers the latest N elements.
● Lists can be used as a message passing primitive.
14

15. Redis SET
In Redis, SETs are similar to LISTs in
that they’re a sequence of strings, but
unlike LISTs, Redis SETs use a hash
table to keep all strings unique.
set-key
item2
item
item3
set
Key name Type of value
Set of distinct values,
undefined order
Command What it does
SADD Adds the item to the set
SMEMBERS Returns the entire set of items
SISMEMBER Checks if an item is in the set
SREM Removes the item from the set, if it exists
15

16. Redis SET implementation details
● Redis SET provides O(1) constant time
regardless of the number of elements
contained inside the set for add, remove and
test for member existance operations.
● Supports server-side operations of unions,
intersections, differences of sets.
● The max number of elements in set 232
- 1.
16

17. redis> SADD set-key item
(integer) 1
redis> SADD set-key item2
(integer) 1
redis> SADD set-key item3
(integer) 1
redis> SADD set-key item
(integer) 0
redis> SMEMBERS set-key
1) "item"
2) "item2"
3) "item3"
Redis SET examples
redis> SISMEMBER set-key item4
(integer) 0
redis> SISMEMBER set-key item
(integer) 1
redis> SREM set-key item2
(integer) 1
redis> SREM set-key item2
(integer) 0
redis> SMEMBERS set-key
1) "item"
2) "item3"
17

18. Redis SET use cases
● Use SETs to track unique things, for
example unique IP addresses or user ids
visiting a web page.
● Sets are good to represent relations
between objects. For example 1 to many
relation could be represented as a set order:
1234:items → [234, 555, 674]
18

19. Redis HASH
Redis HASHes store a mapping of keys to
values. The values that can be stored in
HASHes are strings and numbers.
hash-key
sub-key-1
sub-key-2
hash
Key name Type of value
Distinct keys,
undefined order
Command What it does
HSET Stores the value at the key in the hash
HGET Fetches the value at the given hash key
HGETALL Fetches the entire hash
HDEL Removes a key from the hash, if it exists
Values associated
with the key
value-1
value-2
19

20. Redis HASH implementation details
● The max number of elements in hash 232
- 1.
● A hash with a few fields (where few means
up to one hundred or so) is stored in a way
that takes very little space, so you can store
millions of objects in a small Redis instance.
20

21. redis> HSET hash-key sub-key1 value1
(integer) 1
redis> HSET hash-key sub-key2 value2
(integer) 1
redis> HSET hash-key sub-key1 value1
(integer) 0
redis> HGETALL hash-key
1) "sub-key1"
2) "value1"
3) "sub-key2"
4) "value2"
Redis HASH examples
redis> HDEL hash-key sub-key2
(integer) 1
redis> HDEL hash-key sub-key2
(integer) 0
redis> HGET hash-key sub-key1
"value1"
redis> HGETALL hash-key
1) "sub-key1"
2) "value1"
21

22. ● Hashes are used mainly to represent objects. For
example, storing users as hashes:
Redis HASH use cases
users:1
{
id: 1,
name: john,
email: john@gmail.com
}
users:2
{
id: 1,
name: john,
email: john@gmail.com
}
22

23. Shopping cart example
cart
cart_line
Redis model
SET cart:john → [ 1, 3 ]
SET cart:james → [ 2 ]
HASH cart:1 { user: “john”, status: “submitted” }
HASH cart:2 { user: “james”, status: “in progress” }
HASH cart:1:products [ 28: 1, 372: 2 ]
HASH cart:2:products [ 15: 5, 160: 4 ]
CartID User Status
1 john Submitted
2 james In Progress
3 john Submitted
CartID ProductID Quantity
1 28 1
1 372 2
2 15 5
2 160 4
23

24. Redis ZSET
ZSET (Sorted set) is a data type which is similar to a mix
between a Set and a Hash. Every element (called
member) in a sorted set is associated with a floating point
value (called score).
zset-key
member1
member2
zset
Key name Type of value
Named
members,
ordered by
associated
score
Command What it does
ZADD Adds member with the given score to the
ZSET
ZRANGE Fetches the items in the ZSET from their
positions in sorted order
ZRANGEBYS
CORE
Fetches items in the ZSET based on a range
of scores
ZREM Removes the item from the ZSET, if it exists
Scores, ordered
by numeric value
728
982
24

25. Redis ZSET implementation details
● ZSET supports add, remove, or update elements in a time
proportional to the ln(N) (logarithm of the number of
elements).
● Since elements are taken in order get ranges by score or by
rank (position) could be done in a very fast way.
● Accessing the middle of a sorted set is also very fast, so
ZSETs can be used as a smart list of non repeating elements
where you can quickly access everything you need: elements
in order, fast existence test, fast access to elements in the
middle.
25

26. redis> ZADD zset-key 728 member1
(integer) 1
redis> ZADD zset-key 982 member0
(integer) 1
redis> ZADD zset-key 982 member0
(integer) 0
redis> ZRANGE zset-key 0 -1
withscores
1) "member1"
2) "728"
3) "member0"
4) "982"
Redis ZSET examples
redis> ZRANGEBYSCORE zset-key 0
800 withscores
1) "member1"
2) "728"
redis> ZREM zset-key member1
(integer) 1
redis> ZREM zset-key member1
(integer) 0
redis> ZRANGE zset-key 0 -1
withscores
1) "member0"
2) "982"
26

27. Redis ZSET use cases
● A leader board in an online game, where
every time a new score is submitted you
update it using ZADD.
● Top users can be easily get using ZRANGE.
● Using ZRANK can get a rank by username.
27

28. Redis key rules/recommendations
● Very long/short keys are not a good idea. For example,
key user:1000:followers is more readable than u1000flw.
● While short keys will obviously consume a bit less
memory, your job is to find the right balance between long
and short keys.
● Stick with a schema. For instance object-type:id is a
good idea, as in user:1000. Dots or dashes are often used
for multi-word fields, as in comment:1234:reply.to or
comment:1234:reply-to.
28

29. Redis keys anti-pattern
● We are building a blog and we need to store
comments added by users with keys look
like: posts:post_id:comment_id
● To find all comments added to specific post
you will use command: KEYS posts:1234:*
● Is there a better solution?
29

30. Redis keys anti-pattern (solution)
● The better solution is to use a hash with a new key
scheme posts:post_id:comments
● To add new comments to specific post use next
commands:
○ HSET posts:1234:comments 1 '{"id":1,
"account": 1233, "subject": "..."}'
○ HSET posts:1234:comments 2 '{"id":2,
"account": 1233, "subject": "..."}'
30

31. Redis keys anti-pattern (solution)
● To get all comment ids use command:
○ HKEYS posts:1234:comments
● To get comment by specific id use command:
○ HGET posts:1234:comments 2
● To delete a specific comment use command
○ HDEL posts:1234:comments 2
● To delete all post comments use command
○ DEL posts:1234:comments
31

32. Redis keys expiration
● Redis allows to set timeout (TTL) on keys by calling
EXPIRE key seconds command
● Timeout is cleared only when key is removed or using
DEL or overwritten with SET or GETSET commands
● Expiration could be refreshed by calling EXPIRE
command
● After timeout has expired, the key will be deleted
automatically
32

33. How Redis expires keys?
● Active way
○ Key is actively expired when some client tries to access
it, and the key is found to be timed out.
● Passive way
○ Redis does 10 times per second:
■ Test 20 random keys from the set of keys with an
associated expire.
■ Delete all the keys found expired.
■ If more than 25% of keys were expired, start again
from step 1.
33

34. Redis keys expiration examples
redis> SET mykey "Hello"
OK
redis> EXPIRE mykey 10
(integer) 1
redis> TTL mykey
(integer) 10
redis> SET mykey "Hello World"
OK
redis> TTL mykey
(integer) -1
34

35. Redis Transactions
● All commands are serialized and executed sequentially
● Transactions are atomic (without another client’s command
being executed halfway through)
● Either all commands or no commands will be executed
● Redis commands for transactions:
○ WATCH
○ MULTI
○ DISCARD
○ EXEC
○ UNWATCH
35

36. Redis Transactions example
redis> MULTI
OK
redis> INCR foo
QUEUED
redis> INCR boo
QUEUED
redis> EXEC
1) (integer) 1
2) (integer) 1
redis> WATCH foo, boo
OK
redis> MULTI
OK
redis> SET foo 2
QUEUED
redis> SET boo 3
QUEUED
redis> EXEC
1) OK
2) OK
36

37. Optimistic locking using CAS
● WATCH is used to provide a check-and-set (CAS)
behavior to Redis transactions.
● WATCHed keys are monitored in order to detect
changes against them.
● If at least one watched key is modified before the
EXEC command, the whole transaction aborts, and
EXEC returns a Null reply to notify that the
transaction failed.
37

38. Errors inside transactions
During a transaction it is possible to encounter two kind of
command errors:
● A command may fail to be queued, so there may be an
error before EXEC is called. For instance: syntactically
wrong command or another critical condition
● A command may fail after EXEC is called, for instance
since we performed an operation against a key with the
wrong value (like calling a list operation against a string
value).
38

39. Redis does not support roll backs
● Redis commands can fail only if called with a
wrong syntax, or against keys holding the wrong
data type.
● Redis is internally simplified and faster because
it does not need the ability to roll back.
39

40. Redis Publish/Subscribe
● Redis has first-class support for publishing messages
and subscribing to channels implemented by
SUBSCRIBE, PUBLISH and UNSUBSCRIBE
commands.
● Messages published into channels, without knowledge
of what subscribers there may be.
● Subscribers express interest in one or more channels,
and only receive messages that are of interest, without
knowledge of what publishers there are.
40

41. Pattern-matching subscriptions
● The Redis Pub/Sub supports pattern-
matching subscriptions
● PSUBSCRIBE tweets.* will receive
message from both tweets.john and
tweets.sean channels
● PUNSUBSCRIBE tweets.* will unsubscribe
the client from this pattern
41

42. Redis Pub/Sub example (DEMO)
Subscriber 1
SUBSCRIBE foo
Subscriber 2
SUBSCRIBE foo boo
Publisher
PUBLISH foo “hello”
PUBLISH boo “world”
42

43. Request/Response protocol and RTT
● Redis request is accomplished with the following two
steps:
○ The client sends a query to the server, and reads from
the socket, usually in a blocking way, for the server
response.
○ The server processes the command and sends the
response back to the client.
● RTT (Round Trip Time) - time for the packets to travel
from the client to the server, and back from the server to
the client to carry the reply.
43

44. What’s wrong with blocking request
● Four commands sequence is something like this:
○ Request: INCR X
○ Response: 1
○ Request: INCR X
○ Response: 2
○ Request: INCR X
○ Response: 3
○ Request: INCR X
○ Response: 4
● RTT time is 250 ms, Redis server is able to process 100k
request per second, so will be able to process max 4
requests per second
44

45. Redis Pipelining
● Pipelining is a way to send multiple commands to the server
without waiting for the replies at all, and finally read the
replies in a single step.
● Pipelined version of four commands sequence:
○ Request: INCR X
○ Request: INCR X
○ Request: INCR X
○ Request: INCR X
○ Response: 1
○ Response: 2
○ Response: 3
○ Response: 4
45

46. Redis Pipelining recommendations
● While client sends commands using
pipelining, server forced to queue the
responses in memory.
● To reduce memory usage when sending a
lot of commands better to send them as
batches having a reasonable number.
46

47. Links and Literature
● redis.io/documentation
● The Little Redis Book by Karl Seguin
● Redis in Action by Josiah L. Carlson
● Seven databases in seven weeks (The
Pragmatic Bookshelf, 2012)
47

48. Thank You!
48