Pierre Mavro from Criteo discussed Couchbase usage at their company. Criteo has over 100 Couchbase clusters storing over 90TB of data serving up to 25 million queries per second. They benchmarked Couchbase and found network bandwidth and replicas increased latency. To improve, Criteo monitored latency, split workloads across clusters, automated operations, and tuned Couchbase and systems. Their changes helped Couchbase scale for Criteo's large workload.

1. Couchbase usage and perfs
Criteo - Couchbase Live 2016 - Paris

2. About me
Pierre Mavro - Lead DevOps - NoSQL Team
Working at Criteo as Site Reliability Engineer
@deimosfr

3. Criteo
31 Offices
2000+ employees

4. Criteo technical insights
● 700 engineers
● 17K servers
● 27K displays per second
● 2.4M requests per second

5. Criteo SRE: biggest challenges
● Scaling
● Low latency
● High throughput
● Resiliency
● Automation

6. Couchbase figures at Criteo (Worldwide)
● 1300+ physical servers
● 100+ clusters (up to 50 servers each)
● 90TB of data in memory
● 25M QPS
● < 8ms constant latency

7. Couchbase usage at Criteo
● Storing UUIDs < 30b
● Storing blobs (ex. binary images)
● Storing keys size > value data size (sometimes)
● Serving between 100Kqps to 2.5Mqps per cluster
● Low latency at 99perc < 2ms
● Data size per cluster between 500Gb to ~12Tb (with replica)
● All data fits in memory
● Inter datacenter replication (custom client driver)

8. What we wanted to solve

9. Legacy infrastructure
● Couchbase v1.8 legacy (80%) and v3.0.1
community (20%)
● Slow rebalance (up to 48h for 1 server)
● Rebalance failures on high loaded
clusters
● Max connection reached on v1.8 (9k)

10. Legacy infrastructure
● Same cluster shares on persisted and non
persisted buckets
● No dedicated latency monitoring tool
● No auto restart/upgrade orchestrator
● Server benchmarks update required
● Lack of Couchbase best practices

11. How we achieved the change

12. 1. Benchmarks

13. Benchmarks
● Couchbase Enterprise 3.1.3
● 3x HP GEN9 DL360 (256GB RAM, 6x400GB SSD RAID10, 1Gb Network
interface) (2x injectors + 1 server)
● Key size: UUID string (36 bytes) + Couchbase metadata (56 bytes)
● Value size: uniform range between 750 B and 1250 B (avg 1 kB)
● Number of items: 50M/node (with replica) or 100M/node (without replica)
● Resident active items (= items fully in RAM): ~50%
● Value-only ejection mode (only data value can be removed from RAM, keeping
metadata + key in RAM).

14. Benchmarks
Heavy Writes/Little Reads (10Kqps) without replica
Write rate
per node
Status Disk Write
Queue
Latency 50
perc
Latency 95
perc
Latency
99perc
Latency
99,9 perc
40 Kset/s OK 10M items 0.4 ms 0.7 ms 2 ms 8 ms
60 Kset/s OK 30M items 0.4 ms 0.7 ms 2 ms 20 ms
80 Kset/s OK 50M items 0.4 ms 2 ms 7 ms 30 ms
100 Kset/s OK 70M items 1.5 ms 5 ms 10 ms 40 ms

15. Benchmarks
Heavy Writes/Little Reads (10Kqps) with one replica
Write rate
per node
Status Disk Write
Queue
Latency 50
perc
Latency 95
perc
Latency
99perc
Latency
99,9 perc
20 Kset/s OK 12M items 0.4 ms 1 ms 2 ms 10 ms
30 Kset/s OK 33M items 0.5 ms 2 ms 4 ms 20 ms
40 Kset/s OK 60M items 0.6 ms 2 ms 5 ms 25 ms
50 Kset/s NOK (OOM) >70M items 0.7 ms 5 ms 50 ms 75 ms

16. Benchmarks
Heavy Reads/Little Writes (10Kqps) with one replica
Read rate
per node
Status Disk Write
Queue
Latency 50
perc
Latency 95
perc
Latency
99perc
Latency
99,9 perc
25 Kset/s OK 130k items 0.4 ms 0.7 ms 4 ms 8 ms
50 Kset/s OK 130k items 0.4 ms 1 ms 5 ms 10 ms
75 Kset/s OK 130k items 0.4 ms 5 ms 15 ms 25 ms
100 Kset/s NOK 50k to 500k
items
16 ms 25 ms 45 ms 100 ms

17. Benchmarks
Conclusion for a single node:
● Network 1Gb is the bottleneck
● Replicas introduce latency
● Reads are fast
● Max write with replica per node: 40 Kqps
● Max read with replica per node: 90Kqps
● Max read/write without replica per node: 90 Kqps

18. 2. SLI, SLO & SLA

19. Metrics
Metrics are greats !
● QPS total (read + write)
● Total RAM usage
● Availability
● Number of items
● …
But it’s not relevant enough to know the global service status !

20. SLI: add the major missing metric
Adding latency monitoring as SLI, to be part of our Couchbase SLO and SLA

21. 3. Couchbase support

22. Support contract
● Get latest Couchbase bug fixes
● Suggest Couchbase enhancements
● Speed up resolution of incidents with the help of support
● Get better Couchbase tuning recommendations for performance

23. 4. Refactoring infrastructure

24. Split usages
● High-load (QPS) buckets are on dedicated
clusters
● Low-load (QPS) buckets are shared on
separate “shared” clusters
● Persisted and Non persisted clusters are not
on the same servers anymore

25. 5. Administration Automation

26. Automation: why?
● Need to upgrade from the community to the enterprise version
● Need to apply new configuration options that require a restart of all the
nodes in a cluster
● Need to apply fixes that require a reboot of all the nodes in a cluster
● Need to reinstall servers from scratch

27. Automation: how?
● Criteo is using Chef to bootstrap servers, deploy applications and
configuration
● We did not want to add another new tool in the loop
● Nothing with the required features already exists
● We developed a FOSS Chef cookbook for this and other use-cases:
Choregraphie
https://github.com/criteo-cookbooks/choregraphie

28. Automation: Choregraphie
With Choregraphie we can perform:
● Rolling restart with rebalance
● Rolling upgrade with rebalance
● Use an optional, additional server to speed up rebalance
● Rolling reboot with rebalance
● Rolling reinstall with rebalance
Choregraphie is open source! Feel free to contribute

29. 6. Couchbase and system tuning

30. Couchbase best practices / system tuning
● Minimize swap usage:
○ vm.swappiness = 0 (set to 1 for kernel >3.5)
● Disable transparent Hugepages:
○ chkconfig disable-thp on
● Set SSD IO-Scheduler to deadline:
○ echo “deadline” > /sys/block/sdX/queue/scheduler
● Change CPUFreq governor:
○ modprobe cpufreq_performance
● Leverage maximum connection:
○ max_conns_on_port_XXXX: 30000

31. Couchbase tuning
● Upgrade Nonio parameter to 8 to avoid rebalance failures on high-load
clusters:
○ curl -i -u <Administrator>:<pwd> --data 'ns_bucket:update_bucket_props("<bucketname>",
[{extra_config_string, "max_num_nonio=<N>"}]).' http://<NodeIP>:8091/diag/eval
● Disable access log if you don’t need them to reduce disk usage (native in
Couchbase 4.5):
○ curl -i -u <Administrator>:<pwd> --data 'ns_bucket:update_bucket_props("<bucketname>",
[{extra_config_string, "access_scanner_enabled=false"}]).' http://<NodeIP>:8091/diag/eval

32. Tuning...what’s next?
● Network teaming 802.3ad (bonding) with 2x1Gb cards
● 10Gb network cards
● Upgrade to Couchbase 4.5
● Upgrade kernel to a newer LTS vanilla to enable specific SSD enhancement
(multi queues SSD)
● Switch to Mesos to reduce administration time

33. Questions ?
Criteo - Couchbase Live 2016 - Paris
Pierre Mavro / @deimosfr