1. UC Berkeley - CS10

2. Giv ing a talk at Ca l’s CS10 - The
Beauty and Joy of Computing.
inst. eecs.berkeley.e du/~cs10/sp11
16 Mar via Twitter for iPhone
from UC Berkeley
277 Cory Hall
Berkeley, CA
View Tweets at this place

3. Follow me!
twitter.com/raffi
or send a SMS to 40404
follow raffi

4. To start... What’s a Tweet?

5. Twitter is
the nature of news.

6. “Rudimentary communication
among individuals in real time
allows many to move together as
one” - @biz

7. One Word
April 10, 2008
4:33:23 PM

8. One Word
April 10, 2008
4:33:23 PM
And it’s over
April 11, 2008
1:27:26 PM

9. Super bowl
February 2011
the Grammys
February 2011

10. Tunisia
January 2011
Egypt
January 2011

11. http://twitter.com/#!/toptweets/status/12483108178

12. http://twitter.com/#!/Emergency_In_SF/status/29440739442

13. http://twitter.com/#!/i80chains/status/9726084734

14. http://twitter.com/#!/remedyoakland/status/29002198672

15. http://twitter.com/#!/AlbionsOven/status/10015063036

16. How does Twitter work?

17. Twitter is just three things:
Tweets,relationships between those three
And the users, anD Timelines

18. A Tweet is a simple thing
140 characters

19. What’s in a tweet?
{
“id” : 787167620,
“text” : “Free”,
“date” : “2008-04-1 1 13:27:26”
“user” : “jamesbuck”
}*
*not quite, but close!

20. Where do they go?
Following
Followed by

21. Tweets multiply

22. TWEETs MULTIPLY

23. HOME TIMELINE
[27886578700,
27856404785,
27727644786,
27679412331,
27594550593,
27583976236,
27580246968,
27576503025,
27545926564,
27545450496,
27539864336...

24. Is that it? ... YES! ... sorta ...

25. things get more complicated at scale
And scaling “real-time” is hard

26. How many Tweets are there?

27. How many Tweets are there? 140M!
off the chart

28. 140M tweets 1600 tweets
per day ≈ per second

32. 140M tweets 1B tweets
1600 tweets
per day ≈ per second
week

34. Tweets multiply
8,080 people
need to see my
tweet

35. @ladygaga
mother mons†er
6.8 million followers
@justinbieber
Justin Bieber
5.7 million followers
@BarackObama
44th President of the United States
5.6 million followers
@raffi
me!
5.1 thousand followers

36. @ladygaga
mother mons†er
8.8 million followers
@raffi
me!
8.0 thousand followers
differs by10
scaling 10 63

37. 106deliveries 6000 deliveries
per minute ≈ per millisecond

38. A System is only as strong as its
weakest link

39. 80 MB/sec
≈ 10 ms/seek

40. 10 ms 100 seeks
per seek ≈ per second

41. MySQL
Can’t generate IDs fast enough
Centralized and a single point of failure
snowflake
Highly available and uncoordinated (10kqps)
Compatible with the ecosystem
http://github.com/twitter/snowflake

42. How big are they?
≈
1 tweet text =
≈ 140 characters
18 MB/min
≈ 200 bytes
≈ 26 GB/day
Just tweet text!

43. Lots of deliveries

44. 1
Digraph 2
Need to represent this
4
1 2 3 4 3
1
Matrix
2
Naïve implementation is not scalable
3
4

45. Photo used under Creative Commons from jurvetson
Distributed graph database
flockdb High rate of CRUD operations
Complex set arithmetic queries
http://github.com/twitter/flockdb

46. 200M registered users
2006 2008 2010 2011

47. Using the system
10B API calls 110,000 calls
per day ≈ per second

48. REST API
XML/JSON API over HTTP
Poll-based system / pseudo real-time
hosebird
Streaming API
Long poll HTTP
Near real-time delivery of Tweets

49. Latency
200ms
100ms
0ms

51. Where do we want to be?
Today - 200M people generate ~1600 TPS
Tomorrow - we want to support half the world and all its devices
(right now, there are 6B people and 5B phones)

52. Follow me at
Questions? twitter.com/raffi