These are the slides that the Redis Labs team had used to accompany the session that we gave during the first ever Redis Developers Day on October 2nd, 2014, London. It includes some of the ideas we've come up with to tackle operational challenges in the hyper-dense, multi-tenants Redis deployments that our service - Redis Cloud - consists of.

1. @Redis Developers Day 2014

2. 1st talk: day to day challenges
● ±150 new databases every day (>60,000 total)
● A mix of very small (few MB) and extremely large (TB+) datasets
● PAYG model that can grow to any size by changing maxmemory
● ±250 node failure events (~10 every month) – w/o upgrades
● 5-6 complete data-center outages so far
● Supporting/managing DBs with zero knowledge of application
● Running a modified version of Redis from day 1

3. 2nd talk: Extending Redis to Flash
● Redis #1 hindering factor is the cost of deployment ➔
RAM costs x10 SSD and x250 HDD
● Advanced flash technologies:➔ 1M IOPS (4KB)
➔ Latency: DIMM – 1-2μsec; Plugable – 100μsec

4. Why do we modify Redis?
● Dense multi-tenant architecture and still maintaining
SLAs (single-tenant, dedicated instances supported)
● Efficient large-scale system administration
● Clustering (in production since 06/2012)

5. How do we manage changes?
● Many custom modifications are of no use for anyone else, so we
don’t bother
● Fixes, minor changes to upstream ASAP
● Periodic batches of more significant features
o Low hit rate, community view vs. enterprise/service provider
view
● Our “playground” fork https://github.com/redislabs/redis

6. Skin-of-our-teeth replication
● DB is growing, sync to a bigger instance
● Lots of writes, output buffers and COW bloat
● Redis has a point of no return
⇓
Off-load some of this to the slave
Flush output buffer to slave, Mux RDB

7. Improved Replication Performance
Master
Memory
Master Disk Network Slave Disk Slave
Memory
● Eliminate disk impact on replication performance (pain point)
● Current Redis scheme can still be better sometimes

8. Lua Replication
● Old debate: SCRIPT vs Commands
o In some cases performance is dramatic
o We think both options are needed (user-controlled)

9. External RDB processing
● We do it often, and not only us:
https://github.com/sripathikrishnan/redis-rdb-tools
● Python is too slow for serious work
● We want to avoid code duplication
● Built-in rdb.c was too hard-coded
● Alternative SAX-like (callback-based) parser
● Maybe even librdb.a?

10. Resource Management
● maxmemory can be confusing (think eviction/OOM because of
output buffers)
● How big can we expect the actual process? Master switches are
missing (e.g. max allocated network buffers)
● Graceful handling of output buffer limits
● Fragmentation issues
o jemalloc and the 12% magic
o active defrag and bigger ideas for bigger problems

11. More Resource Management
● Automatic process oom_score_adj settings to guard
data in low memory events
● Automatically reject BGSAVE and BGREWRITEAOF or
terminate child on low memory

12. Cluster features
● Lua key ranges / patterns
● Global keys

13. Extending Redis to Flash - goals
● More (cheap) memory not better persistence
● 100% compatibility with Redis
● Pluggable storage back-ends / non-volatile DIMM
● Utilize storage concurrency
● Use case: limited set of hot objects over large keyspace
● Different from other approaches: Ledis, ARDB, SSDB

14. Extending Redis to Flash - blocks
Main thread
Clients
I/O job
queues
I/O
Threads
K/V storage
backend API
Storage
backend

15. Extending Redis to Flash - moar
● Still being prepared for prime time
● Planned to be open sourced once ready

16. Thank You