1. Cassandra
@Formspring
Tuesday, August 30, 2011
2. Cassandra
@Formspring
Yet another Knight, Dragon and Princess Story.
Tuesday, August 30, 2011
3. Formspring helps people find out more about each
other by sharing interesting & personal responses
Tuesday, August 30, 2011
30. MySQL + Memcache
• Flat MySQL table
• Indexes (Followers vs Following)
Tuesday, August 30, 2011
31. MySQL + Memcache
• Flat MySQL table
• Indexes (Followers vs Following)
• Memcache
Tuesday, August 30, 2011
32. Memcache
• Stored as JSON list
Tuesday, August 30, 2011
33. Memcache
• Stored as JSON list
• Can get out of sync
Tuesday, August 30, 2011
34. MySQL
• Hundreds of millions of rows
Tuesday, August 30, 2011
35. MySQL
• Hundreds of millions of rows
• m2.4xl EC2 instance
Tuesday, August 30, 2011
39. QOTD
• Feb: 35K Followers
Tuesday, August 30, 2011
40. QOTD
• Feb: 35K Followers
• Mid-Feb: 100K
Tuesday, August 30, 2011
41. QOTD
• Feb: 35K Followers
• Mid-Feb: 100K
• Growing about 30K a week.
Tuesday, August 30, 2011
42. QOTD
• Feb: 35K Followers
• Mid-Feb: 100K
• Growing about 30K a week.
• 200K: closer to Memcache 1MB limit.
Tuesday, August 30, 2011
43. QOTD
• Feb: 35K Followers
• Mid-Feb: 100K
• Growing about 30K a week.
• 200K: closer to Memcache 1MB limit.
• (with gz compaction . . .)
Tuesday, August 30, 2011
48. Redis
• Key : JSON blob of IDs
Tuesday, August 30, 2011
49. Redis
• Key : JSON blob of IDs
• ~2-3s / insert
Tuesday, August 30, 2011
51. Frontend Ask
Outbox Mem
QW
MySQL
Cache
Tuesday, August 30, 2011
52. Frontend Ask
SELECT account_id
FROM follow
Outbox WHERE following = 1234 Mem
QW
MySQL
Cache
Tuesday, August 30, 2011
53. Frontend Ask
Outbox OH SHIT Mem
QW
MySQL
Cache
Tuesday, August 30, 2011
54. Frontend Ask
Outbox OH SHIT Mem
QW
MySQL
Cache
Tuesday, August 30, 2011
55. Frontend Ask
Outbox OH SHIT Mem
QW
MySQL
Cache
Gazillion messages
Tuesday, August 30, 2011
56. Frontend Ask
Outbox OH SHIT Mem
QW
MySQL
Cache
Inbox
Inbox
Inbox
Gazillion messages
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QW
QW
QWs
Tuesday, August 30, 2011
57. Frontend Ask
Outbox OH SHIT Mem
QW
MySQL
Cache
Inbox
Inbox
Inbox
Gazillion messages
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QW
QW
Inbox
QWs
Object
Tuesday, August 30, 2011
58. Frontend Ask
Outbox
*HANGING*
OH SHIT Mem
QW
MySQL
Cache
Inbox
Inbox
Inbox
Gazillion messages
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QW
QW
Inbox
QWs
Object
Tuesday, August 30, 2011
59. Frontend Ask
Outbox
*OOM*
OH SHIT Mem
QW
MySQL
Cache
Inbox
Inbox
Inbox
Gazillion messages
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QW
QW
Inbox
QWs
Object
Tuesday, August 30, 2011
60. Frontend Ask
Outbox
Outbox
Outbox
Outbox OH SHIT Mem
Outbox
QW
Outbox
QW MySQL
Outbox
QW
QW
QW
QW
QW
Cache
Inbox
Inbox
Inbox
Gazillion messages
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QW
QW
Inbox
QWs
Object
Tuesday, August 30, 2011
61. Frontend Ask
Outbox
Outbox
Outbox
Outbox OH SHIT Mem
Outbox
QW
Outbox
QW MySQL
QW
OOM
QW
QW
QW Cache
Inbox
Inbox
Inbox
Gazillion messages
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QWInbox
Inbox
QW
QW
QW
Inbox
QWs
Object
Tuesday, August 30, 2011
62. We need a solution
Tuesday, August 30, 2011
84. Build a Social Graph
• Python
• Cassandra (Obviously..)
Tuesday, August 30, 2011
85. Build a Social Graph
• Python
• Cassandra (Obviously..)
• PyCassa
Tuesday, August 30, 2011
86. Build a Social Graph
• Python
• Cassandra (Obviously..)
• PyCassa
• Thrift
Tuesday, August 30, 2011
87. 4 types of actions
• Following
Tuesday, August 30, 2011
88. 4 types of actions
• Following
• Followers
Tuesday, August 30, 2011
89. 4 types of actions
• Following
• Followers
• Blocks
Tuesday, August 30, 2011
90. 4 types of actions
• Following
• Followers
• Blocks
• Blockers
Tuesday, August 30, 2011
111. It’s OK if it fails.
Tuesday, August 30, 2011
112. Remember
idempotence is your
friend.
Tuesday, August 30, 2011
116. Frontend Ask
Cassandra
Cluster
Tuesday, August 30, 2011
117. Frontend Ask
Outbox
Cassandra
QW
Cluster
Tuesday, August 30, 2011
118. Frontend Ask
Outbox
Cassandra
QW
Cluster
Iterative
KW
Tuesday, August 30, 2011
119. Frontend Ask
Outbox
Cassandra
QW
Cluster
0, N)
ser_id,
ers(u
get_follow
Iterative
KW
Tuesday, August 30, 2011
120. Frontend Ask
Outbox
Cassandra
QW
Cluster
’, ..., ‘ 1999’]
‘1234 , ‘1238
[
Iterative
KW
Tuesday, August 30, 2011
121. Frontend Ask
Outbox
Cassandra
QW
Cluster
’, ..., ‘ 1999’]
‘1234 , ‘1238
[
Iterative
KW
N Messages
Tuesday, August 30, 2011
122. Frontend Ask
Outbox
Cassandra
QW
Cluster
’, ..., ‘ 1999’]
‘1234 , ‘1238
[
Iterative
KW
N Messages
Inbox
QW
Tuesday, August 30, 2011
123. Frontend Ask
Outbox
Cassandra
QW
Cluster
’, ..., ‘ 1999’]
‘1234 , ‘1238
[
Iterative
KW
Inbox N Inbox
QW Object
Tuesday, August 30, 2011
124. Frontend Ask
Outbox
Cassandra
QW
Cluster
+N)
ser_i d, N, N
u
wers(
get _follo
Iterative
KW
Inbox N Inbox
QW Object
Tuesday, August 30, 2011
127. Reduce our #instances
12x 12x
Inbox
Inbox Inbox
Inbox
Inbox
Inbox Inbox
Inbox
QWs
QWsInbox
Inbox QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox QWs Inbox
Outbox
QWs
QWs
QWs QWs
QWs
QWs
QWs QWs
QWs
Tuesday, August 30, 2011
128. Reduce our #instances
12x 4x
Inbox
Inbox
Inbox
Inbox
QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox
QWs
QWsInbox
Inbox Inbox
Inbox
QWs
QWsInbox
Inbox Inbox
Inbox
QWs
QWs Outbox
QWs
QWs
QWs
QWs QWs
QWs
QWs
Tuesday, August 30, 2011
129. Reduce our #instances
12x 4x
Inbox
Inbox
Inbox
QWsInbox
QWsInbox
QWsInbox
QWsInbox
QWsInbox Inbox
QWsInbox
QWsInbox Inbox
Inbox
QWsInbox Inbox
Outbox
QWs
Inbox
QWs Inbox
QWsInbox QWs
QWs
QWsInbox
Inbox QWs
QWs
QWs+
QWs
QWs
QWs
Iterator
QW
Tuesday, August 30, 2011
133. Counting
• Fast
Tuesday, August 30, 2011
134. Counting
• Fast
• Not-accurate
Tuesday, August 30, 2011
135. Counting
• Fast
• Not-accurate
• Faking “Read-repair”
Tuesday, August 30, 2011
140. Cassandra allowed us:
• To create new features.
• To scale.
• To save money.
Tuesday, August 30, 2011
141. Cassandra allowed us:
• To create new features.
• To scale.
• To save money.
• To sleep. (Almost there)
Tuesday, August 30, 2011
142. Cassandra allowed us:
• To create new features.
• To scale.
• To save money.
• To sleep. (Almost there)
• Lots of fun :)
Tuesday, August 30, 2011
