Redis for your boss 2.0

REDIS: Need for speed
@elena_kolevska
Elena Kolevska 1
About Our Company
REDISFORYOURBOSS
Elena Kolevska

Who am I?
A nomad earthling | Self employed developer
BLOG.ELENAKOLEVSKA.COM
@ELENA_KOLEVSKA

(Really) SHORT
INTRO
No, seriously!

REDISISANOPENSOURCE(BSD licensed),
IN-MEMORYDATASTRUCTURESTORE,USED
ASDATABASE,CACHEANDMESSAGE
BROKER
The 'Definition' on redis.io

• Different data structures
• Keys with a limited time-to-live
• Lua scripting
• Transactions
• Pipelining
• Pub/Sub
• Built-in replication
• Different levels of on-disk persistence
BASIC FEATURES:

AVAILABLE CLIENTS IN:
ActionScript
Common
Erlang
Haxe
Matlab
Pascal
Racket
Smalltalk
bash
Crystal
Fancy
Io
mruby
Perl
Rebol
Swift
C
D
gawk
Java
Nim
PHP
Ruby
Tcl
C#
Dart
GNUProlog
Javascript
Nodejs
PureData
Rust
VB
C++
Elixir
Go
Julia
ObjectiveC
Python
Scala
VCL
Clojure
Emacs lisp
Haskell
Lua
OCaml
R
Scheme

AVAILABLE DATA STRUCTURES:
•STRINGS (Binary safe, can be anything from "hello world" to a jpeg ﬁle)
•LISTS(Collections of string elements sorted according to the order of insertion)
•SETS (Collections of unique, unsorted string elements)
•SORTEDSETS (It's like Sets with a score)
•HASHES (Objects. Maps of ﬁelds associated to values. Think non-nested json objects)
•BITMAPS (Manipulate Strings on a bit level)
•HYPERLOGLOGS (Probabilistic data structure used to estimate the cardinality of a set)

STRINGS
•Binary safe, can be from "hello world" to a jpeg ﬁlekeyname
value •Maximum size = 512MB
•Keynames can be emojis 🐘🦄😀

LISTS
keyname
value1
value2
value3
value2
value5
•Collection of strings
•Sorted by order of insertion
•Great for queues and timelines
•LPOP, BLPOP, LPUSHX, BRPOPLPUSH are O(1) operations
•Max length: > 4 billion elements

SETS
keyname
value1
value2
value3
value4
value5
•Collections of unique, unsorted string elements
•Great for tracking unique appearances
•Unions, intersections, diﬀerences

HASHES
keyname •Maps between string fields and string values
•Great for tracking representing objects
field1
field2
field3
field4
value1
value2
value3
value4

SORTED SETS
keyname •Sets with a floating number score
•Very fast access to elements in the middle
•Perfect for leaderboards
field1
field2
field3
field4
0.6379
6.379
63.79
637.9
•Get ranks of elements
•Get ranges of sorted elements
•Sort by value and by field, indirectly (lex)

BITMAPS
•Manipulate Strings on a bit levelkeyname
value •Set / clear individual bits
•Find the ﬁrst set or unset bit
•Get count of all set/unset bits

HYPERLOGLOGS
•Probabilistic data structure used for estimating cardinalitykeyname
*
•Uses at most 12KB
•Conceptually similar to sets
•1% margin of error
•Used in IOT

I loved the service at #statscompany,
30 minutes from ticket creation to
resoltuion. Well done guys, I'm
impressed.
I hate the service at #statscompany, I've
been on hold for the past 20minutes.

OUR SCENARIO: TWITTER ANALYZER DASHBOARD
751
TOTAL TWEETS
751Main stream
120sub-stream foo
107sub-stream bar
 




...
Substream:keyword 1 Substream:keyword 2
Love
Hate
Great
0 30 60 90 120
86
107
120
Total tweets
Scores
Main Stream

[1]CONNECTINGTOREDIS
Install the PRedis package using Composer
> composer require predis/predis
Initialise the client
$parameters = [
'scheme' => 'tcp',
'host' => '127.0.0.1',
'port' => 6379
];
$client = new PredisClient($parameters, ['prefix' => 'stats:']);

[2]SETTRACKEDDATA
Use sets to store all the hashtags we'll be looking at and all the keywords as well
$client->sadd('hashtags', 'thestatscompany','statscompany',
'statscampaign');
// hashtags | 'thestatscompany'
// | 'statscompany'
// | 'statscampaign'
$client->sadd('keywords', 'loved', 'amazed', 'frustrated',
'problem', 'terrible', 'amazing', 'great', 'hate');

[3]GETTHEDATA
Use Twitter Stream API to receive notifications for tweets containing any of the hashtags we're following
$hashtags = $client->smembers('hashtags');
// array (size=3)
// 0 => string 'thestatscompany' (length=15)
// 1 => string 'statscompany' (length=12)
// 2 => string 'statscampaign' (length=13)

Save every new tweet from the stream as a separate string:
$keyname = 'tweet_id:' . $tweet_id;
$tweet_contents = "Amazed by the customer service of #statscompany";
$client->set($keyname, $tweet_contents);
$keyname = 'tweet_id:' . $tweet_id;
$tweet_contents = "Amazed by the customer service of #statscompany";
$client->set($keyname, $tweet_contents);

And then push to a queue to be processed asynchronously
$client->lpush('message_queue', $keyname);
// 'message_queue' | 'tweet_id:45645656'
// | 'tweet_id:44645234'
// | 'tweet_id:43645232'
$client->lpush('message_queue', $keyname);
// 'message_queue' | 'tweet_id:45645656'
// | 'tweet_id:44645234'
// | 'tweet_id:43645232'

[4] WORKER TO
PROCESS THE
QUEUED JOBS

[4]WORKERTOPROCESSTHEQUEUEDJOBS
A separate worker will be grabbing jobs off the top of the queue and processing them:
$message_queue = $client->rpop('message_queue');$message_queue = $client->rpop('message_queue');
Reliable queue: RPOPLPUSH, BRPOPLPUSH
Blocking queue: BLPOP, BRPOP

[5]PROCESSTHETWEETCONTENT
$tweet_contents = $client->get($keyname);
$keywords = $client->smembers('keywords');
foreach ($keywords as $keyword) {
$tweet_contents = strtolower($tweet_contents);
$keyword = strtolower($keyword);
if (strpos($tweet_contents,$keyword) !== false){
$client->zincrby('mention_counter', 1, $keyword);
$keyword_feed_keyname = 'keyword_feeds:'. $keyword;
$client->lpush($keyword_feed_keyname, $tweet_contents);
$client->ltrim($keyword_feed_keyname, 0, 50);
}
}
$client->incr('total_count'); // Increase the general tweet count
$client->lpush('main_feed', $tweet_contents);
$client->ltrim('main_feed', 0, 100);
}
}

[5]PROCESSTHETWEETCONTENT
}
}

[6]SHOWTHESTATS
$total_count = $client->get('total_count');
// 'total_count' | 259
$scores = $client->zrevrangebyscore('mention_counter', '+inf', '-inf', ['withscores'=>1]);
// mention_counter | 'amazed' => 9.00
// | 'frustrated' => 5.00
// | 'love' => 4.00
// Feed by keyword
foreach ($scores as $keyname => $score) {
$keyword_feeds[$keyname] = $client->lrange('keyword_feeds:' . $keyname, 0, -1);
}
// Feed of all tweets containing one of the specified hashtags
$main_feed = $client->lrange('main_feed', 0, -1);

Redis is TCP server using client/server model. A request to the server is
accomplished in two steps:
•The client sends a query to the server, and reads from the socket, usually in a blocking way,
for the server response
•The server processes the command and sends the response back to the client
Client Server
Networking link
Request from client to server
Response from server to client

Redis is a TCP server using the client/server model. A request to the server
is accomplished in two steps:
•The client sends a query to the server, and reads from the socket, usually in a blocking way,
for the server response
•The server processes the command and sends the response back to the client
Client Server
Request from client to server
Response from server to client
RTT: Round Trip Time

Number of
requests to
server
•Save the tweet
•Add tweet to main feed
•Increase counter for total tweets
•Get all monitored keywords
•Check tweet for any keywords
•Increase counter for found keywords
•Add tweet to sub-feed
•Update score
9THESTEPS:

Number of
requests to
server
•Save the tweet
•Add tweet to main feed
•Increase counter for total tweets
•Get all monitored keywords
•Check tweet for any keywords
•Increase counter for found keywords
•Add tweet to sub-feed
•Update score
13?
} + 4 requests for every found keyword
THESTEPS:

1
Number of
requests to
server</php
$keywords = $client->smembers("keywords");
$responses = $client->pipeline(function ($pipe) use ($keywords) {
$pipe->set(...); // Save the tweet
$pipe->lpush(...); // Save the tweet to main feed
$pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever
$pipe->incr(...); // Increase the counter for the keyword
$pipe->lpush(...); // Add to the subbed
$pipe->ltrim(...); // Trim the stream to X tweets
$pipe->zincrby(...); // Update score
}
});
USINGTHEPREDISLIBRARYFORPHP

2
Number of
requests to
server</php
$keywords = $client->smembers("keywords");
$responses = $client->pipeline(function ($pipe) use ($keywords) {
$pipe->set(...); // Save the tweet
$pipe->lpush(...); // Save the tweet to main feed
$pipe->ltrim(...); // Trim the main feed to X tweets so it doesn't grow forever
$pipe->incr(...); // Increase the counter for the keyword
$pipe->lpush(...); // Add to the subfeed
$pipe->ltrim(...); // Trim the stream to X tweets
$pipe->zincrby(...); // Update score
}
});
USINGTHEPREDISLIBRARYFORPHP

Create a hello.lua ﬁle:
return "Hello"
> redis-cli --eval /path/to/script/hello.lua
"Hello"

hello.lua:
local tweet = ARGV[1]
local key_tweet = KEYS[1]
local key_keywords = KEYS[2]
local key_total_tweets_count = KEYS[3]
local key_scores = KEYS[4]
local key_main_feed = KEYS[5]
redis.call("SET", key_tweet, tweet) -- Save the tweet
redis.call("INCR", key_total_tweets_count) -- Increase the total tweet count
redis.call("LPUSH", key_main_feed, tweet) -- Push the tweet to the main feed
redis.call("LTRIM", key_main_feed, 0, 100) -- Trim the main feed
local keywords = redis.call("SMEMBERS", key_keywords) -- Get the keywords
for i, name in ipairs(keywords) do
if string.find(tweet, name) then
local substream_name = "sub_feed:" .. name
redis.call("LPUSH", substream_name, tweet) -- Push the tweet to the sub feed
redis.call("LTRIM", substream_name, 0, 100) -- Trim the sub feed
redis.call("ZINCRBY", key_scores, 1, name) -- Increment the score in the leaderboard
end
end
return "OK"

> redis-cli --eval hello.lua tweet_123 keywords ... , "A tweet about the words foo and bar"
"OK"
1!
Number of
requests to
server

SCRIPT
•Evaluates a script cached on the server side by its SHA1 digest. The command is otherwise
identical to EVAL.
> redis-cli SCRIPT LOAD "$(cat hello.lua)"
"6d52847f03028ab1d4620b60dd6ef4a14c8727d7"
> redis-cli evalsha 6d52847f03028ab1d4620b60dd6ef4a14c8727d7 5
> tweet_123 keywords ... "A tweet about the words foo and bar"
"OK"
EVALSHA
•Load a script into the scripts cache, without executing it. Returns a SHA-1 digest of the script.

•Dynamically loaded libraries
•Written in C
•Almost as fast as the Redis core
•Let you extend Redis commands, create new data structures, access data almost
as fast as native Redis commands
•Add-ons to Redis
•Coming in version 4.0, currently in Beta (RC)
WHATAREREDISMODULES?

•Low-level: Close to native access to core data structures
•High-level: Client-like access to core and modules' commands
LAYERSOFTHEMODULESAPI

RedisModule_Call(ctx,"INCR","sc",argv[1],"10");
Command
Format specifier
Arguments
The context object
c -- Null terminated C string pointer.
b -- C buffer, two arguments needed: C string pointer and size_t length.
s -- RedisModuleString as received in argv or by other Redis module APIs returning a RedisModuleString object.
l -- Long long integer.
v -- Array of RedisModuleString objects.
! -- This modifier just tells the function to replicate the command to slaves and AOF. It is ignored from the point of view of
arguments parsing.
HIGHLEVELAPI

Anatomy of a module
#include "redismodule.h"
#include <stdlib.h>
int ProcessTweet_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// ...
}
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
// Register the module or error out
if (RedisModule_Init(ctx, "tweet_processor", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;
// Register the command or error out
if (RedisModule_CreateCommand(ctx, "tweet.process", ProcessTweet_RedisCommand,
"readonly", 1,1,1) == REDISMODULE_ERR) return REDISMODULE_ERR;
return REDISMODULE_OK;
}
Deﬁnitions of the API
Called whenever the module is loaded
Must be present in each Redis module
Register the module or error out
Register the command or error out
Command Function name

Anatomy of a module
#include <stdlib.h>
// ...
}
if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand,
}
The logic of the module

How much faster does it get?
Anatomy of a module
#include <stdlib.h>
// ...
}
if (RedisModule_Init(ctx, "process_tweet", 1, REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, "process_tweet.rand", ProcessTweet_RedisCommand,
}
// We need EXACTLY 5 arguments
if (argc != 5) return RedisModule_WrongArity(ctx);
RedisModule_AutoMemory(ctx);
RedisModuleCallReply *reply;
reply = RedisModule_Call(ctx, "INCR", "s", argv[3]);
RedisModule_ReplyWithCallReply(ctx, reply);
}
Enable automatic memory management
Call Redis commands
Reply with a call object

RedisModule_AutoMemory(ctx);
RedisModuleKey *key;
key = RedisModule_OpenKey(ctx,argv[1],REDISMODULE_READ|REDISMODULE_WRITE);
if (RedisModule_KeyType(key) == REDISMODULE_KEYTYPE_EMPTY) {
RedisModule_StringSet(key,argv[2]);
}
RedisModule_CloseKey(key);
Open a key. Return a key pointer
Checks the type of the key
Set a string value for a key
Close the key
LOWLEVELAPI

Compiling
$ gcc -fPIC -std=gnu99 -c -o process_tweet process_tweet.c
$ ld -o process_tweet.so process_tweet.o -shared -Bsymbolic -lc
On Linux:
$ gcc -dynamic -fno-common -std=gnu99 -c -o process_tweet.o process_tweet.c
$ ld -o process_tweet.so process_tweet.o -bundle -undefined dynamic_lookup -lc
On OSX:
Loading
./redis-unstable/src/redis-server --loadmodule ./process_tweet.so

Calling the command
127.0.01:6379> tweet.process tweet_id:42 "This is my tweet about the words foo and bar" ...
Our command

Redis for your boss 2.0

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Redis for your boss 2.0

Similaire à Redis for your boss 2.0 (20)

Dernier

Dernier (20)

Redis for your boss 2.0