3. http://twitpic.com/135xa There’s a plane in the
Hudson. I’m on a ferry
going to pick up the
people. Crazy.
!
15 Jan 09
Janis Krums @jkrums
@raffi
qcon shanghai 2013
4. what are the goals?
⇢ evolve from being solely a web stack
@raffi
qcon shanghai 2013
16. what are the goals?
⇢ evolve from being solely a web stack
⇢ isolate responsibilities and concerns
⇢ site speed and reliability
⇢ developer innovation speed
@raffi
qcon shanghai 2013
43. Write API
Blender
HTTP Push
Mobile
Push
Hadoop
Batch Compute
Redis
Redis
Redis
Timeline Cache
Search Cache
Redis
Redis
Earlybird
Fanout
Push Compute
Ingester
Social
Graph
Service
Timeline
Service
@raffi
qcon shanghai 2013
44. Write API
⇢ pipelined 4k
“destinations”
at a time
Blender
⇢ replicated
HTTP Push
Mobile
Push
Hadoop
Batch Compute
⇢ keyed off
“recipient”
Redis
Redis
Redis
Timeline Cache
Redis
Redis
Earlybird
Search Cache
insert
Fanout
Push Compute
Ingester
Social
Graph
Service
Timeline
Service
@raffi
qcon shanghai 2013
45. Write API
Ingester
Blender
⇢ RPUSHX to
only add to
cached
timelines
Tweet IDPush User ID
HTTP
8 bytes
8 bytes
Mobile
Push
Bits
Hadoop
4 bytes
Batch Compute
⇢ native list
structure
Redis
Redis
Redis
Push Compute
Redis
Redis
Earlybird
Search Cache
using redis
Timeline Cache
Fanout
Timeline
Service
@raffi
qcon shanghai 2013
46. Write API
Ingester
Blender
⇢ RPUSHX to
only add to
cached
timelines
Tweet IDPush User ID
HTTP
Bits
Tweet ID
Bits Hadoop
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
Timeline
Service
Mobile
Tweet ID
User ID
Push
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Batch Compute
⇢ native list
structure
Redis
Redis
Redis
Push Compute
Redis
Redis
Earlybird
Search Cache
using redis
Timeline Cache
Fanout
Bits
@raffi
qcon shanghai 2013
47. Write API
Ingester
Blender
⇢ RPUSHX to
only add to
cached
timelines
Tweet IDPush User ID
HTTP
Bits
Tweet ID
Bits Hadoop ID
Tweet
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
Timeline
Service
Mobile
Tweet ID
User ID
Push
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Bits
Tweet ID
User ID
Batch Compute
⇢ native list
structure
Redis
Redis
Redis
Push Compute
Redis
Redis
Earlybird
Search Cache
using redis
Timeline Cache
Fanout
Bits
Tweet ID
Tweet ID
@raffi
qcon shanghai 2013
75. Hosebird
Write API
Hosebird
Firehose
User Streams
Hosebird
Track / Follow
event propagation
⇢ write API sends all events into hosebird;
sees content creation events, social graph
changes, etc.
⇢ different queues for public tweets,
protected tweets, social events, etc.
@raffi
qcon shanghai 2013
81. Hosebird
Write API
Hosebird
Firehose
Track / Follow
Hosebird
User Streams
track / follow
⇢ simple query based on tweet content
⇢ keeps list of terms / users of interest
⇢ parses public tweets at the edge, and if
term matches a token, or user is of
interest, then route
@raffi
qcon shanghai 2013
83. Hosebird
Write API
Hosebird
Firehose
Track / Follow
Hosebird
User Streams
user streams
⇢ replicate home timeline experience
⇢ upon login, obtain “following” list
⇢ keep cached following list coherent by
seeing social graph updates
⇢ route tweet if from a followed user
@raffi
qcon shanghai 2013
131. User Intent
Query Expansion
“Hello, world”
“Hello” AND “world”
@raffi’s home timeline
home_timeline:raffi
OR
user_timeline:taylorswift13
@raffi
qcon shanghai 2013
132. streaming compute
⇢ continuous computation
⇢ driven by the events that come into
twitter
⇢ generalizing the push mechanism
@raffi
qcon shanghai 2013