Packets per second (PPS) is an often overlooked value within an environment. Most network concerns are around throughput and interface speed, but what happens when this value becomes the bottleneck due to large-scale hosting providers (AWS, Azure, etc.) with rigid standards? This talk covers what a Redis packet looks like and how the out-of-the-box configuration can drastically affect packet per second overhead. From here we’ll deep dive into specific configuration values which help lower PPS numbers, as well as different Redis master/slave relationships that can be utilized to keep PPS below inflexible network thresholds.

1. For Lovers of Packets and Jumbo Frames
Redis Networking Nerd Down

2. Intro
John Bullard
VP Engineering
Benji Taylor
DevOps Lead

3. We Block Bots
Distil

4. How the Distil Bot Detection Solution Works
As web traffic passes through Distil, the system
1. Fingerprints each incoming connection and
compares it to our Known Violators Database
1. If it’s a new fingerprint, validates the browser
to determine if it’s a Bot or Not
1. Based on your preferences, automatically tags,
challenges, or blocks the bot

5. Bad Bots Cause Problems

6. Fortune 500 & Alexa Global 10,000 Customers
Ecommerce
Travel
Publishers
Directories
Traditional Media
Marketplace
Services

7. We Block Bots
Distil Architecture

8. Distil Architecture

9. Topology - Single Master

10. Topology - Master : Master

11. Topology - Master : Slave

12. Every Packet Matters
Customer Story

13. Customer Story
○ Large Online Marketplace
High traffic volume
Approaching seasonal peak - 3x traffic
AWS EC2 Classic
○ Soft 100k PPS Limit
○ Customer concern

14. Customer Story
○ Scaling Up + Out
Master : Slave Topology
○ 9 Node: 1 Master, 8 Slaves
C3.8xlarge Instances
○ 36 vCPU, 60 GiB, 10 Gigabit
Proof of Concept
○ Onboard <5% of traffic

15. POC - Slave Measurements

16. POC - Master Measurements

17. POC - Master Measurements

18. Customer Story

19. Customer Story

20. Customer Story
○ Bottleneck - Master PPS
Redis Master : Write + Replicate
○ All nodes write
○ All writes replicate to all nodes
○ Requirements
Master PPS < 100k
No partitioning
EC2 Classic
$0 Cost Increase

21. We Block Bots
Packets

23. Packets

24. Packets

25. Packets

26. Packets

27. Overview
○ Jumbo Caveats
Not supported everywhere (Virtualbox)
“Jumbo” differs among vendors
Every component on path must support it
Switches - drop jumbo frames
Routers - must support ingress and egress

28. Overview
○ Tuning - Jumbo frames -
AWS Support for jumbo frames:
○ Compute optimized: C3, C4, CC2
○ General purpose: M3, M4, T2 - GPU: CG1, G2
○ Memory optimized: CR1, R3
○ Storage optimized: D2, HI1, HS1, I2
MTU 9001 bytes

29. Packets

30. Packets

31. Packets

32. Packets

33. Packets
○ TCP Small Packet Problem
Application sending 1 byte packets
TCP packets have a 40 byte header
○ 20 bytes for TCP
○ 20 bytes for IPv4
41 byte packet
○ For 1 byte of useful information

34. Packets
○ Nagle’s algorithm (RFC 896)
Combine small outgoing messages
Reduce # of packets
Improve TCP/IP efficiency
Trade-off: increased latency
○ Problematic for real-time systems
○ Per-socket: TCP_NODELAY

35. Packets
# Disable TCP_NODELAY on the slave socket after SYNC?
#
# If you select "yes" Redis will use a smaller number of TCP packets and
# less bandwidth to send data to slaves. But this can add a delay for
# the data to appear on the slave side, up to 40 milliseconds with
# Linux kernels using a default configuration.
#
# If you select "no" the delay for data to appear on the slave side will
# be reduced but more bandwidth will be used for replication.
#
# By default we optimize for low latency, but in very high traffic conditions
# or when the master and slaves are many hops away, turning this to "yes" may
# be a good idea.
repl-disable-tcp-nodelay no

36. Packets

37. Packets

38. Packets
Master/Slave Chains
Common Architecture for Read Intensive Scenarios
Balance Write Duplication to Read-Slaves
Shift PPS to Read-Slaves
Configuration Change - No Dev
No Added Cost
○ 1 Master
2 Read-Slaves
6 Edgenodes

39. Customer Story

40. Customer Story

41. Packets
○ Approach
Ethernet tuning - Enable Jumbo Packets
TCP/IP tuning - Enable “Naggling”
Topology - Read-Slave fan-out

42. Demo