6. PRESENTED BY
• 6+ years in production
• 1M+ database created
• Running on over 500+ data-centers/zones
• 3000+ node failure events
• 100+ complete data-center outages
Redis Enterprise production deployment history
7. PRESENTED BY
• 6+ years in production
• 1M database created
• Running on over 500+ data-centers/zones
• 3000+ node failure events
• 100+ complete data-center outages
Redis Enterprise production deployment history
8. PRESENTED BY
The reliability ingredients of a database
Data-PersistenceHA + Failover Backup & Restore Active-Active
9. PRESENTED BY
The reliability ingredients of a database
Data-PersistenceHA + Failover Backup & Restore Active-Active
10. PRESENTED BY
HA Concept #1 – pure in-memory replication
Disk-based Replication
(OSS default)
M R
OSS Diskless Replication
M S
Pure In-Memory Replication
M S
1
2
3 1 2
1
11. PRESENTED BY
HA Concept #1 – pure in-memory replication
Disk-based Replication
(OSS default)
M R
OSS Diskless Replication
M R
Pure In-Memory Replication
M S
1
2
3 1 2
1
12. PRESENTED BY
HA Concept #1 – pure in-memory replication
Disk-based Replication
(OSS default)
M R
OSS Diskless Replication
M R
Pure In-Memory Replication
M R
1
2
3 1 2
1
x2 faster
16. PRESENTED BY
HA Concept #3 – how to deploy a multi-az/rack cluster
(1) At least 3 zones
A B
C
17. PRESENTED BY
HA Concept #3 – how to deploy a multi-az/rack cluster
M3
R3
R2M2
R1
M1
(2) Master and replica of each shard should be deployed on a different zone
A B
C
18. PRESENTED BY
HA Concept #3 – how to deploy a multi-az/rack cluster
M3
R3
R2M2
R1
M1
(3) Distance between zones < 10msec
A B
C
<10msec
<10msec
<10msec
19. PRESENTED BY
HA Concept #3 – how to deploy a multi-az/rack cluster
M3
R3
R2M2
R1
M1
(4) The majority of the node shouldn’t be deployed on one zone
A B
C
27. PRESENTED BY
• Application level solution → too complex to write
• LWW (Last Write Wins) → doesn’t work for many of the Redis data-types,
e.g.:
Conflict Resolution is hard
Lists
BitmapsBit field
Sorted SetsSets
Counters
32. PRESENTED BY
Monthly Uptime Percentage Service Credit Percentage
Active-Active SLA
26.30sec downtime/mo
Less than 99.999%, but equal
to or greater than 99%
10%
Less than 99% 25%
Multi-AZ SLA
4.38min downtime/mo
Less than 99.99%, but equal
to or greater than 99%
10%
Less than 99% 25%
Standard SLA
43.83min downtime/mo
Less than 99.9%, but equal to
or greater than 99%
10%
Less than 99% 25%
Redis (Enterprise) Cloud – 99.999% uptime
39. PRESENTED BY
App
App
App
Node 1
Node 2
Node M
Linearly scalable architecture
Minimalinter-node/Rediscommunication
A single network hop, multi-core, shared nothing architecture
Proxy 1 Shard 1 Shard 2 Shard n
Proxy 2 Shard n+1 Shard n+2 Shard 2n
Proxy M Shard n*(m-1)+1 Shard n*mShard n*(m-1)+2
40. PRESENTED BY
What we showed @ Redisconf18
20M ops/sec 30M ops/sec 50M ops/sec
1.92M – per node
97.65K – per shard
§
5,020,000
11,380,000
21,120,000
30,200,000
41,190,000
50,420,000
3 6 12 18 24 26
ops/sec
# of nodes
Cluster Throughput (@ 1 msec Latency)
41. PRESENTED BY
What we have done since Redisconf18
Redis 5.0 includes multiple performance improvements
(e.g. big pipeline)
Improved proxy performance & proxy<>shard communication
m4.16xlarge (64 vCore @ 2.4GHz) c5.18xlarge (72 vCores @ 3.5GHz)
NUMA binding and CPU affinity
42. PRESENTED BY
Shards 4 6 8 8 10
Proxy threads 16 24 28 28 32
NUMA setting Default Default Default Tuned Tuned
Latency (msec) 0.92 0.91 0.98 0.81 0.92
Throughput (M ops/sec) 2.9 3.89 4.2 4.8 5.3
Throughput per shard
(K ops/sec)
725 648 525 600 530
Optimal performance on a single c5.18xlarge instance
SET:GET ratio – 1:1; Pipeline -9; Value size – 100B
43. PRESENTED BY
How to tune Redis Enterprise with NUMA and CPU affinity
Proxy threads affinity
Redis shards CPU/Memory bind to node0
Proxy threads affinity
Redis shards CPU/Memory bind to node1
45. PRESENTED BY
200M+ ops/sec on 40 only c5.18xlarge instances
5,350,000
51,720,000
102,370,000
201,170,000
1 10 20 40
ops/sec
# of nodes
Cluster Throughput (@<1 msec latency)
46. PRESENTED BY
Close to optimal linear scaling - 94%
5,350,000
51,720,000
102,370,000
201,170,0005,350,000 5,172,000 5,118,500
5,029,250
3,400,000
3,900,000
4,400,000
4,900,000
5,400,000
5,900,000
0
50,000,000
100,000,000
150,000,000
200,000,000
250,000,000
1 10 20 40
Nodeops/sec
Clusterops/sec
# of nodes
Cluster and Node Throughput (@ < 1msec latncy)
47. PRESENTED BY
Breaking our own record (big time!) in just 15 months!
March 2018 June 2019 Improvement
Cluster throughput 50M ops/sec 200M ops/sec x4
# of nodes 26 40 x2.6
# of shards 512 400 x5
51. PRESENTED BY
DRAM = 768GB
SSD = 3,072GB
DRAM = 768GB
On DRAM
1M+ ops/sec, $$$
On Flash/SSD
100K+ ops/sec, $
Paying just +15%
for
x5 memory capacity
Running Redis on multiple memory technologies
52. PRESENTED BY
1TB replicated dataset (2TB in total) @ 100K ops/sec
<1msec <1msec <10msec <1msec
presented by
$725,217.00
$507,652
$418,108
$145,745
$0.00
$100,000.00
$200,000.00
$300,000.00
$400,000.00
$500,000.00
$600,000.00
$700,000.00
$800,000.00
ElastiCache OSS Redis DynamoDB Redis Enterprise (RoF)
Redis by Cloud
Providers
OSS Redis Other NoSQL Redis on Flash
53. PRESENTED BY
DRAM = 768GB
SSD = 3,072GB
DRAM = 768GB
On DRAM
1M+ ops/sec, $$$
On Flash/SSD
100K+ ops/sec, $
Running Redis on multiple memory technologies
Can Redis run
1M+ ops/sec
on a single server
with multi-TB dataset
@ affordable cost
?
54. PRESENTED BY
DRAM = 768GB
SSD = 3,072GB
On Flash/SSD
100K+ ops/sec, $
On DRAM
1M+ ops/sec, $$$
DRAM = 768GB
On Persistent Memory
1M+ ops/sec, $$
Redis Enterprise on multiple memory technologies
PMEM = 3,840GB
55. PRESENTED BY
DRAM = 768GB
SSD = 3,072GB
On Flash/SSD
100K+ ops/sec, $
On DRAM
1M+ ops/sec, $$$
DRAM = 768GB
On Persistent Memory
1M+ ops/sec, $$
Redis Enterprise on multiple memory technologies
PMEM = 7680GB
56. PRESENTED BY
56
Same Redis performance at lower cost and higher capacity
DRAM
Optane DC Persistent
Memory system
2.41M ops/sec 0.89msec
0.87msec2.26M ops/sec
65. PRESENTED BY
Can RedisTimeSeries be as fast and efficient as Redis?
The benchmark:
• 4000 devices
• Each sends 10 CPU
metrics every 10 sec for
3 hours, total 10,800
metrics
• Server –1xc5.18xlarge
• Client - 1xc5.18xlarge
9,879,392.87
11,548,460.21
32,479,407.31
2,338,343.30
Redis Streams
RedisTimeSeries
Sorted-Set per Device
Sorted-Set per Metric
Metrics ingest/sec
631
124,324
808
1,896
Redis Streams
RedisTimeSeries
Sorted-Set per Device
Sorted-Set per Metric
Queries/sec
1.54
3.27
5.33
40.75
Redis Streams
RedisTimeSeries
Sorted-Set per Device
Sorted-Set per Metric
GB Used
84. PRESENTED BY
RedisGears – high-performance architecture
GearsCoordinator MapReducerGearsExecuter
Gears infrastructure is written in C
C - API
85. PRESENTED BY
Stream processing example
Build a gear that creates a timeseries of the number of messages that come into the
stream every second
RedisTimeSeriesRedis Streams
Every sec
86. PRESENTED BY
# create a pipe of operations
gb = GearsBuilder('StreamReader’)
# Increment a timeseries counter for this second
gb.foreach(lambda x: execute('TS.INCRBY', 'msg_count', 1, 'RESET', 1))
# register the execution on `stream`
gb.register(‘stream’)
Build a gear that creates a timeseries of the number of messages that come into the
stream every second
Stream processing example
91. PRESENTED BY
Robotics arm use case
1000/sec Sensors
data
Filtered
data
Commands
Streams2TS TS2AI AI2Streams
92. PRESENTED BY
Robotics arm use case
1000/sec Sensors
data
EDGE
Filtered
data
Commands
Streams2TS TS2AI AI2Streams
93. PRESENTED BY
Q1 Is Redis reliable ? 99.999% availability
Q2 How fast can Redis go ? 200M ops/sec @ < 1msec latency on only 40 nodes
cluster
Q3 Is Redis affordable 80% savings with Redis on Flash, Intel PM as a DRAM
replacements
Recap
Q4 What else Redis can do ? RedisTimeSeries GA; multi-model operation
@ < 1msec, everywhere (cloud, on-prem, edge)