Slides from the Basho webinar held on

1. Load-Testing &
Benchmarking Riak
Sean Cribbs Jon Meredith
Developer Advocate Developer
basho

2. Why Benchmark?
basho

3. Why Benchmark?
•Minimize surprises during deployment
basho

4. Why Benchmark?
•Minimize surprises during deployment
•Test expected load
basho

5. Why Benchmark?
•Minimize surprises during deployment
•Test expected load
•Test peak load
basho

6. Why Benchmark?
•Minimize surprises during deployment
•Test expected load
•Test peak load
•Assess headroom
basho

7. Why Benchmark?
•Minimize surprises during deployment
•Test expected load
•Test peak load
•Assess headroom
•Find failure points
basho

8. Why Benchmark?
•Minimize surprises during deployment
•Test expected load
•Test peak load
•Assess headroom
•Find failure points
•You must measure to know!
basho

9. Types of Benchmarks
basho

10. Types of Benchmarks
•Throughput
basho

11. Types of Benchmarks
•Throughput
•operations / sec
basho

12. Types of Benchmarks
•Throughput
•operations / sec
•systems experience
basho

13. Types of Benchmarks
• Throughput May be multiple
requests!
•operations / sec
•systems experience
basho

14. Types of Benchmarks
• Throughput May be multiple
requests!
•operations / sec
•systems experience
•Latency
basho

15. Types of Benchmarks
• Throughput May be multiple
requests!
•operations / sec
•systems experience
•Latency
•time per operation
basho

16. Types of Benchmarks
• Throughput May be multiple
requests!
•operations / sec
•systems experience
•Latency
•time per operation
•user experience
basho

17. Types of Benchmarks
• Throughput May be multiple
requests!
•operations / sec
•systems experience
•Latency
•time per operation Need mean,
•user experience median, percentiles
basho

18. Benchmarking
Overview
basho

19. Benchmarking
Overview
•Determine your application’s needs
basho

20. Benchmarking
Overview
•Determine your application’s needs
•Test typical and worst-case scenarios
basho

21. Benchmarking
Overview
•Determine your application’s needs
•Test typical and worst-case scenarios
•Minimize variables changed between
scenarios
basho

22. Benchmarking
Overview
•Determine your application’s needs
•Test typical and worst-case scenarios
•Minimize variables changed between
scenarios
•Run early, run often
basho

23. Benchmarking Steps
basho

24. Benchmarking Steps
•Start up your test cluster
basho

25. Benchmarking Steps
•Start up your test cluster
•Configure a test
basho

26. Benchmarking Steps
•Start up your test cluster
•Configure a test
•Run the test against the cluster
basho

27. Benchmarking Steps
•Start up your test cluster
•Configure a test
•Run the test against the cluster
•Generate pretty graphs (requires R)
basho

28. Benchmarking Steps
•Start up your test cluster
•Configure a test
•Run the test against the cluster
•Generate pretty graphs (requires R)
•Interpret graphs, tweak, re-test
basho

29. basho_bench
basho

30. basho_bench
•Dave “Dizzy” Smith’s experience testing
instant messaging and database systems
basho

31. basho_bench
•Dave “Dizzy” Smith’s experience testing
instant messaging and database systems
•Created for internal benchmarking
basho

32. basho_bench
•Dave “Dizzy” Smith’s experience testing
instant messaging and database systems
•Created for internal benchmarking
•Benchmark anything key/value-like
basho

33. basho_bench
•Dave “Dizzy” Smith’s experience testing
instant messaging and database systems
•Created for internal benchmarking
•Benchmark anything key/value-like
•Used heavily on innostore and bitcask
basho

34. basho_bench
•Dave “Dizzy” Smith’s experience testing
instant messaging and database systems
•Created for internal benchmarking
•Benchmark anything key/value-like
•Used heavily on innostore and bitcask
•Simple, extensible Erlang API
basho

35. basho_bench
basho

36. worker worker worker worker
basho_bench
basho

37. Event Generator
get update
80% 20%
worker worker worker worker
basho_bench
basho

38. Event Generator
get update
80% 20%
worker worker worker worker
Driver (HTTP, Protobuffs, etc)
basho_bench
basho

39. Event Generator
get update
80% 20%
Key
Generator
uniform
worker worker worker worker integer
Driver (HTTP, Protobuffs, etc)
basho_bench
basho

40. Event Generator
get update
80% 20%
Key
Generator
uniform
worker worker worker worker integer
Driver (HTTP, Protobuffs, etc)
Value
Generator
fixed
8KB
basho_bench
basho

41. Event Generator
get update
80% 20%
Key
Generator
uniform
worker worker worker worker integer
Driver (HTTP, Protobuffs, etc)
Value
Generator
fixed
Riak 8KB
basho_bench
basho

42. Demo
basho

43. Microbenchmarks
Are Bad
basho

44. Microbenchmarks
Are Bad
•Benchmarks should be long-running
basho

45. Microbenchmarks
Are Bad
•Benchmarks should be long-running
•Cache warmup
basho

46. Microbenchmarks
Are Bad
•Benchmarks should be long-running
•Cache warmup
•Page flushes
basho

47. Microbenchmarks
Are Bad
•Benchmarks should be long-running
•Cache warmup
•Page flushes
•RAM vs. Disk
basho

48. Microbenchmarks
Are Bad
•Benchmarks should be long-running
•Cache warmup
•Page flushes
•RAM vs. Disk
•Test the limits of the infrastructure
basho

49. basho

50. Key Generation
•sequential_int_bin
•uniform_int_bin
•pareto_int_bin
basho

51. Uniform Distribution
{uniform_int_bin, 10000}
CDF Uniform
1.0
0.8
• Equal chance
0.6
of selecting
cdf
any key 0.4
0.2
0.0
0 2000 4000 6000 8000 10000
key
basho

52. Pareto Distribution
{pareto_int_bin, 10000}
1.0
0.8
• Hard coded so
ppareto(x, 2000, 1.5)
0.6
20% of keys
receive 80%
0.4
of requests 0.2
0.0
0 2000 4000 6000 8000 10000
x
basho

53. Value Generation
•fixed_bin
•exponential_bin
basho

54. Exponential Distrib.
{exponential_bin, 5000, 2500}
Exponential CDF
1.0
• Value size:
0.8
Min + Exp(Lambda)
0.6
P(Size <= size)
• Set Lambda 0.4
1 / (Mean - Min)
0.2
0.0
0 5000 10000 15000 20000
size
basho

55. Benchmarking is Hard
basho

56. Benchmarking is Hard
•Design an accurate test
basho

57. Benchmarking is Hard
•Design an accurate test
•Tool and system limits
basho

58. Benchmarking is Hard
•Design an accurate test
•Tool and system limits
•Testing a multi-variate space
basho

59. Benchmarking is Hard
•Design an accurate test
•Tool and system limits
•Testing a multi-variate space
•Easy to take results out of context
basho

60. Benchmarking is Hard
•Design an accurate test
•Tool and system limits
•Testing a multi-variate space
•Easy to take results out of context
•Everything is relative!
basho

61. Gotchas
basho

62. Gotchas
•Running out of file handles
basho

63. Gotchas
•Running out of file handles
•Swapping / thrashing
basho

64. Gotchas
•Running out of file handles
•Swapping / thrashing
•Running benchmark on same machine
basho

65. Conduct your own
basho

66. Conduct your own
•Take metrics from existing app
basho

67. Conduct your own
•Take metrics from existing app
•Mixture of get/put/delete operations
basho

68. Conduct your own
•Take metrics from existing app
•Mixture of get/put/delete operations
•Value size distribution
basho

69. Conduct your own
•Take metrics from existing app
•Mixture of get/put/delete operations
•Value size distribution
•Key distribution
basho

70. Conduct your own
•Take metrics from existing app
•Mixture of get/put/delete operations
•Value size distribution
•Key distribution
•“Hot” and “cold” keys
basho

71. Conduct your own
•Take metrics from existing app
•Mixture of get/put/delete operations
•Value size distribution
•Key distribution
•“Hot” and “cold” keys
•Configure a test, run, RINSE & REPEAT
basho

72. Plug
Interested in learning about support,
consulting, or Enterprise features?
Email info@basho.com or go to
http://www.basho.com/contact.html to talk
with us.
www.basho.com
basho