4. What is Atomicity?What is Atomicity?
Series of database operations
Guaranteed to all run or none run
Prevents operations from running partially
The effects of all operations are immediately visible
I.e. another client cannot see partial state
Also referred to as a Transaction
5. How do these tools work?How do these tools work?
Redis is Mostly Single-Threaded
Except for things like Background IO
Node.js is Mostly Single-Threaded
Except for IO and Node v10 Worker Threads
Single Client & Server is simple
Things get complicated with multiple clients
12. Atomic, Multi-Operation CommandsAtomic, Multi-Operation Commands
Common use-cases have single-command variants
INCR key # GET key ; ~value++ ; SET key ~value
SETNX key value # !EXISTS key ; SET key value
LPUSHX key value # EXISTS key ; LPUSH key value
RPOPLPUSH src dest # RPOP src ; LPUSH dest ~value
GETSET key value # GET key ; SET key value
13. INCRINCR is an Atomic Incrementis an Atomic Increment
Scenario: Two clients want to increment counter
14. INCRINCR is an Atomic Incrementis an Atomic Increment
Client #1 atomically increments value of counter
15. INCRINCR is an Atomic Incrementis an Atomic Increment
Client #2 atomically increments value of counter
17. PipeliningPipelining
Ensures commands are run in order per-connection
Sends a batch of commands separated by newlines
Commands are sent in the same message
The Node.js redis module usually does this anyway
18. Pipelining: Example CodePipelining: Example Code
redis.batch()
.zrangebyscore('jobs', 0, now) // get jobs
.zremrangebyscore('jobs', 0, now) // delete jobs
.exec((error, data) => {
let jobList = data[0];
console.log('jobs', jobList); // perform work
});
ZRANGEBYSCORE jobs 0 1000rnZREMRANGEBYSCORE jobs 0 1000
19. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
It looks atomic
Prevents command interleaving on one connection
A subset of commands can fail
Other client pipelines can interleave commands
20. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
const redis = require('redis').createClient();
// run 10 instances of this process in parallel
let keys = [];
for (let i = 0; i < 100000; i++) {
keys.push(i);
}
shuffle(keys);
let pipeline = redis.batch();
for (let key of keys) {
pipeline = pipeline.hsetnx('pipeline', `k${key}`, process.pid);
}
pipeline.exec(() => redis.quit());
22. Pipelining: What's it for?Pipelining: What's it for?
Reducing network latency
Send several commands in one message
Receive several responses in one message
echo "PINGrnPINGrnPINGrn" | nc localhost 6379
+PONG
+PONG
+PONG
24. MULTI: True AtomicityMULTI: True Atomicity
Atomic regardless of other clients / connections
Client sends MULTI , more commands, EXEC
Other clients can still run commands
Queued commands are run sequentially
Any failures and the entire transaction fails
25. MULTI: Example CodeMULTI: Example Code
redis.multi()
.zrangebyscore('jobs', 0, now) // get jobs
.zremrangebyscore('jobs', 0, now) // delete jobs
.exec((error, data) => {
let jobList = data[0];
console.log('jobs', jobList); // perform work
});
MULTI
ZRANGEBYSCORE jobs 0 1553099335332
ZREMRANGEBYSCORE jobs 0 1553099335332
EXEC
26. MULTI Drawback: No command chainingMULTI Drawback: No command chaining
Can't use command result as argument
E.g., cannot pop from list, assign to new key
28. Lua: The Ultimate in AtomicityLua: The Ultimate in Atomicity
There's a simpler, less-efficient EVAL command
Send the entire script every time
Like sending a normal SQL query
Or use SCRIPT LOAD ahead of time
Then use EVALSHA to run code via resulting hash
Like executing a SQL Stored Procedure
Declare key names as arguments for sharding
29. Lua: Server-Side LogicLua: Server-Side Logic
Output of one command can be piped into another
Other processing can happen, too
30. Lua: Game Lobby Example CodeLua: Game Lobby Example Code
-- add-user.lua: add user to lobby, start game if 4 players
local lobby = KEYS[1] -- Set
local game = KEYS[2] -- Hash
local user_id = ARGV[1] -- String
redis.call('SADD', lobby, user_id)
if redis.call('SCARD', lobby) == 4 then
local members = table.concat(redis.call('SMEMBERS',lobby),",")
redis.call('DEL', lobby) -- empty lobby
local game_id = redis.sha1hex(members)
redis.call('HSET', game, game_id, members)
return {game_id, members}
end
return nil
32. Lua: DrawbacksLua: Drawbacks
Another language to maintain
Simple language, easy syntax
Increases overhead on Redis server
An infinite loop could lock up server
Need to load scripts before using to be efficient
It's idempotent; load scripts when app starts
33. RecapRecap
Executing singular commands are atomic
Executing multiple commands are not atomic
Pipelining is not atomic, but it's fast
MULTI is atomic, but you can't chain results
Lua scripts are atomic and chainable
