Slides from San Francisco Redis Meetup, August 6, 2013. Sharding Redis at Flite

1. SHARDING REDIS at FLITE
Eugene Feingold & Chakri Kodali

2. WHAT IS FLITE?
• Display advertising platform
• Create and publish rich,
interactive ads
• Serve ad impressions
• Collect and analyze metrics,
batch AND realtime

3. WHAT IS FLITE?
• Display advertising platform
• Create and publish rich,
interactive ads
• Serve ad impressions
• Collect and analyze metrics,
batch AND realtime
with

4. REALTIME YOU SAY?
• Realtime monitoring of ad performance
• Debugging of ads, with instant feedback on triggered events

5. METRICS ARCHITECTURE
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js
pipelined write of ~30 items

6. WRITING TO REDIS
Persistence: Up to 4 hours for most data
Amount: ~30 writes per event, pipelined
Granularity: By second, minute, hour
Write Types:
• HSET - JSON blobs (Event body data)
• LPUSH - Lists of events (Events by session)
• HINCRBY - Simple counters (Events by ad)
• ZINCRBY - Sorted set counters (To retrieve “Top 100”)
• PUBSUB - Stream event data (Debugger)

7. READING FROM REDIS
Redis transactions are used extensively like multi, exec
Read types:
• HGETALL - get all data for event
• HGET - get event counts by ad
• LRANGE - list of all events by session
• ZREVRANGE - top 100 ads with highest number of events
• SUBSCRIBE

8. AND ALL OF THIS AT SCALE
• Daily traffic peaks: 100k - 200k events per minute
• Peaks are really plateaus that last for hours
• Read load is negligible by comparison, but reads must be fast
• In fact, everything must be fast: <1 sec latency for debugger to work

9. WHAT’S WRONG WITH THIS PICTURE?
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js

10. WHAT’S WRONG WITH THIS PICTURE?
Bottleneck!
JVM
JVM
JVM
ad
ad
ad
ad
ad
ad
ad
node.js
node.js

11. SCALABILITY FAIL!
What did it look like?
2-3x of our usual load

12. SCALABILITY FAIL!
Note how only 1 core is being used 14,000 open connections!
What did it look like?

13. A QUICK SOLUTION?
• MOAR Megahurtz!!: m2.2xl is already about as fast as Amazon gets.
• Redis-As-A-Service: Expensive, not fast enough for even our usual load.
• twemproxy: Twitter’s sharding solution.
Doesn’t support all commands: PING, MULTI, INFO, MGET, etc…

14. WHAT ABOUT JEDIS’S NATIVE SHARDING?
• No pipelining
• No pubsub
• Complicated consistent hashing mechanism makes reading in other
environments more difficult

15. LET’S ROLL OUR OWN!
Goals: Speed, speed, speed
Not Goals: Fault tolerance, redundancy, resiliency

16. HOW HARD CAN IT BE?
Sharding method: Java hashCode of key for every item written
JVMad
event items
node.js
Write to many Read from one

17. WHAT HAPPENED?
Before Sharding After Sharding
Items per Event 30 30
Items written per Event per Redis 30 10
Redis Connections per Event 1 3
Connections per second per Redis box n n
Reality: When n gets to around 500, Redis maxes out CPU and starts rejecting connections.
Theory: Since Redis claims to be able to handle 70k connections per second, the amount of
data being sent per connection is the problem.

18. SO HOW HARD CAN IT BE?
HARD.

19. TAKE TWO
Sharding method: Java hashCode of EVENT key for every item written.
A single key now lives on multiple Redis boxes
JVMad
event items
node.js
Write to one Read from many

20. BETTER!
Before Sharding After Take 1 After Take 2
Items per Event 30 30 30
Items written per Event per Redis 30 10 30
Redis Connections per Event 1 3 1
Connections per second per Redis box n n n/3
More load can be easily accommodated by adding boxes

21. CODING CHALLENGES
Java
• Managing multiple connection pools
• Managing multiple pipelines
• Automatic health checks
node.js
• Finding hashing function that works in different environments
• Managing multiple pipelines
• Fanout requests and merging response once pipeline is
executed

22. SINGLE-REDIS JEDIS WORKFLOW
On application startup:
1. Initialize jedisPool with connection info
Every time:
1. Jedis jedisClient = jedisPool.getClient();
2. Pipeline pipeline = jedisClient.pipelined();
3. State your business
4. pipeline.sync();
5. jedisPool.returnResource(jedisClient);

23. SHARDED JEDIS WORKFLOW
On application startup:
1. Initialize n jedisPools with connection info
Every time:
1. Jedis jedisClient = jedisPool.getClient();
2. Pipeline pipeline = jedisClient.pipelined();
3. State your business
4. pipeline.sync();
5. jedisPool.returnResource(jedisClient);

24. SHARDED JEDIS WORKFLOW
On application startup:
1. Initialize n jedisPools with connection info
Every time:
1. Jedis jedisClient = jedisPool.getClient(); Which pool?
2. Pipeline pipeline = jedisClient.pipelined(); Which client?
3. State your business To whom?
4. pipeline.sync(); Which pipeline?
5. jedisPool.returnResource(jedisClient); Return what where?

25. SHARDED JEDIS WORKFLOW
RedisNodeManager
Spring-created singleton
JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager

26. SHARDED JEDIS WORKFLOW
RedisNodeManager
Spring-created singleton
JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager
RedisPipelineManager JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager
getPipelineManager()

27. SHARDED JEDIS WORKFLOW
RedisNodeManager
Spring-created singleton
JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager
RedisPipelineManager JedisPoolManagerJedisPoolManagerJedisPoolManagerRedisPoolManager
Pipeline
getPipelineManager()
getPipeline(shardKey)

28. LET’S LOOK AT SOME CODE!

29. OPERATIONAL DETAILS
• Redis is single threaded
o You can run multiple Redises on one server
o Bind each Redis instance to a specific core

30. QUESTIONS?