Slides deck for a Redis Conference 2015 session that discusses the lessons learned from benchmarking Redis' performance.

1. Benchmarking Redis
By itself and versus other NoSQL databases
@ItamarHaber #RedisConf 2015

2. About Myself
A Redis Geek and Chief Developers Advocate at
Have you subscribed to the newsletter yet?
https://redislabs.com/redis-watch-archive

3. define:benchmark
"In computing, a benchmark is the act of running a
computer program, a set of programs, or other
operations, in order to assess the relative performance
of an object, normally by running a number of standard
tests and trials against it."
https://en.wikipedia.org/wiki/Benchmark_(computing)

4. The Only Truism About Benchmarking
The chances that any benchmark's results will
accurately reflect the performance of your
specific use case are infinitesimal*
* single exception: benchmarks that you run on
your own application <- please wait patiently for
my guest speaker

5. So Why Benchmark Redis at all?
• Quantify Blazing Fast with some ballpark figures
• Regression testing/optimizations verifications/sanity
• Comparative benchmark(et)ing
• Fun & profit

6. Howto #1: redis-benchmark
• Included with Redis (build it with make)
• Built-in suite of tests and can also run arbitrary
operations (the -t switch)
• The perfect tool to profile Redis (and moar) but...
• Like Redis, it is single-threaded so generating load is
challenging
• Does not report errors - if results are exceptionally high,
there's probably an error in the statement that you're
checking 

7. Howto #2: memtier_benchmark
https://github.com/RedisLabs/memtier_benchmark
• Open source traffic generator for Redis (and Memcached)
• We use it for baseline profiling, stress loading and benchmarking
• Notable differences (compared to redis-benchmark):
• Multi-threaded -> a single client can create considerable loads
• Executes n requests or runs for m seconds
• Generates Read:Write traffic patterns (e.g. 80:20)
• Does uniform and Gaussian distributions
• OTOH – only does GET/SET (or GETRANGE/SETRANGE)

8. Benchmarking With Multiple Clients – Ouch!

9. redisbenchmarkframework
• https://github.com/RedisLabs/redisbenchmarkframework
• Python & Fabric
• Executes redis-benchmark/memtier_benchmark/other and custom
scripts from multiple servers
• Can spawn multiple threads per server
• Collects RPS results and outputs CSV

11. Benchmarking vs. Other NoSQLs, an i.e.g.
Aerospike vs. Redis benchmark
• http://lynnlangit.com/2015/01/28/lessons-learned-benchmarking-nosql-on-
the-aws-cloud-aerospikedb-and-redis/
• http://antirez.com/news/85
• https://redislabs.com/blog/the-lessons-missing-from-benchmarking-nosql-
on-the-aws-cloud-aerospikedb-and-redis
To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains.
"To test Redis doing GET/SET
is like to test a Ferrari
checking how good it is at
cleaning the mirror when it
rains."
Salvatore Sanfilippo

12. Benchmarking vs. Other NoSQLs, i.e.g. #2
Google Compute Engine's How Many Servers Do You Need To Change a Lightbulb 1
Million Writes/Sec Benchmark Series
• DataStax – 300 servers: http://googlecloudplatform.blogspot.com/2014/03/cassandra-
hits-one-million-writes-per-second-on-google-compute-engine.html
• Aerospike – 50 servers: http://googlecloudplatform.blogspot.com/2014/12/aerospike-
hits-one-million-writes-Per-Second-with-just-50-Nodes-on-Google-Compute-Engine.html
<- We actually needed
two, publication
pending

13. "Comparative" Advertising – There Is A War
• The Open Source Redis Project is (and IMO should always remain) pure
• However,
• The users want numbers and comparisons
• The analysts want numbers and comparisons
• The press wants numbers and comparisons
• Redis' competitors are providing numbers and comparisons
• Redis is extremely popular, but it may not remain so without the public's
attention
• The Redis community and ecosystem is (and IMO should continue to)
provide this collateral in form of benchmarks, success stories, case studies,
etc…

14. All Is Fair In Love And War…
And Benchmarking

15. An i.e.g.: Yahoo Cloud Serving Benchmark
"The goal of the Yahoo Cloud Serving Benchmark (YCSB) project is to
develop a framework and common set of workloads for evaluating the
performance of different "key-value" and "cloud" serving stores."
http://labs.yahoo.com/news/yahoo-cloud-serving-benchmark/
YCSB Redis client:
https://github.com/brianfrankcooper/YCSB/blob/master/redis/src/mai
n/java/com/yahoo/ycsb/db/RedisClient.java

16. The YCSB Challenge
• The original YCSB client implementation for Redis
• Uses a Hash for storing multiple fields per key
• Uses Redis' commands correctly (i.e. HMGET)
• … but is still too slow
• "Weapon-grade", modified YCSB Redis client:
• https://gist.github.com/dragnot/06ef5d01df8cb181555a
• Uses GET/SET
• Objects are serialized client-side with MessagePack
• Objects are updated server-side with Lua (cmsgpack)

17. YCSB Results