I’ve learned how to build distributed systems the hard way; I’ve failed, and failed again. I’ve made many of the common mistakes and tried a few other things that turned out to be a disappointment. You shouldn't have to make those mistakes too. In this talk I'll tell the story of how I built a real time advertising analytics platform that tracks and reports on millions of impressions every day, and all the things I did wrong before I got it to work. I’ll also tell you what I did right, and the choices I don’t regret.

1. LEARNING TO BUILD
DISTRIBUTED SYSTEMS
THE HARD WAY
@iconara

2. speakerdeck.com/u/iconara
(real time!)

3. Theo / @iconara

4. Chief Architect at

5. let’s make online advertising a great experience

6. MAKING THIS

7. INTO THIS

8. HOW HARD CAN IT BE?

9. TRACKING
AD IMPRESSIONS
track page views and all their ads
track visibility and send updates on changes
track events, track activity, sync cookies,
and track visits

10. LOADED
VISIBLE
HIDDEN
LOADED
VISIBLE
track page views and all their ads
track visibility and send updates on changes
track events, track activity, sync cookies,
and track visits

11. ASSEMBLING
SESSIONS
assemble ad impressions, page views and visits,
to be able to calculate things like total visible duration
mix in demographics, revenue, and third-party data

12. WAS
HIDDEN
BECAME {
A CLICK! "user_id": "M9L6R5TD0YXK",
ACTIVE "session_id": "MAI3QAGNAIYT",
"timestamp": 1347896675038,
"placement_name": "example",
"category": "frontpage",
"embed_url": "http://example.com/",
"visible_duration": 1340
"browser": "Chrome",
"device_type": "computer",
BECAME BECAME
"click": true,
"ad_dimensions":"980x300"
WAS VISIBLE }
VISIBLE
LOADED
AGAIN
3rd PARTY DATA &
OTHER GOODIES
assemble ad impressions, page views and visits,
to be able to calculate things like total visible duration
mix in demographics, revenue, and third-party data

13. ANALYTICS
precompute metrics, count uniques,
build visitor histories for attribution

14. precompute metrics, count uniques,
build visitor histories for attribution

15. HOW HARD CAN IT BE?

16. 25K REQUESTS
PER SECOND
~1 billion requests per day, 1 TB raw data

17. ONE VISIT CAN
CHANGE UP TO
100K COUNTERS
hundreds of millions of individual counters per day,
plus counting uniques and visitor histories

18. IN REAL TIME
or near real time, if you want to be pedantic

19. START WITH TWO
OF EVERYTHING
going from one to two is the hardest

20. GIVE A LOT OF
THOUGHT TO YOUR
KEYS AND IDS
it will save you lots of pain

21. a timestamp
something random
MANLO0 JME57Z
monotonically increasing,
sorts nicely

22. something random
a timestamp
JME57Z MANLO0
uniformly distributed,
works nicely with sharding

23. PUT BUFFERS
BETWEEN LAYERS
queues can even out peaks,
let you scale layers independently,
and let you restart services without loosing data

24. SEPARATE
PROCESSING
FROM STORAGE
that way you can scale each independently

25. ×
× ××
× ×
×
PLAN HOW TO GET
RID OF YOUR DATA
deleting stuff is harder than you might think

26. ×
NoDB
keep things streaming

27. STREAM
PARTITIONING

28. RANDOMLY
when you have no interdependencies
between things it’s easy to scale out
(or round robin, it’s basically the same)

29. CONSISTENTLY
when there are interdependencies you need
to route using some property of the objects,
but make sure you get a uniform distribution

30. NUMEROLOGY

31. 12

32. 2 | 12
3 | 12
4 | 12
6 | 12

33. 8 | 24
5 | 60

34. 12, 60, 120, 360
superior highly composite numbers

35. 12, 60, 120, 360
superior highly composite numbers

36. 12, 60, 120, 360
superior highly composite numbers

37. 12, 60, 120, 360
superior highly composite numbers

38. 12, 60, 120, 360
superior highly composite numbers

39. 12, 60, 120, 360
superior highly composite numbers

40. 12, 60, 120, 360
superior highly composite numbers

41. 12, 60, 120, 360
superior highly composite numbers

42. for maximal flexibility partition with multiples of 12

43. for maximal flexibility partition with multiples of 12

44. A SHORT
DIVERSION ABOUT
COUNTING TO 60
the reason why there’s 60 seconds to a minute,
and 360 degrees to a circle

45. 3 SEGMENTS
ON EACH FINGER
= 12

46. 3 SEGMENTS
ON EACH FINGER
= 12
FIVE FINGERS
ON OTHER HAND
= 60

47. log2(366) ≈ 31

48. $-$
(ASCII code 36)-----

49. log2(366) ≈ 31

50. log2(366) ≈ 31
six characters 0-9, A-Z can represent 31 bits,
which is kind of almost very close to four bytes

51. MANLO0

52. Time.now.to_i.to_s(36).upcase
MANLO0
a timestamp

53. DO YOU REALLY
NEED A BACKUP?
if you got 3x replication over multiple
availability zones, is that backup really worth it?

54. PRODUCTION IS THE
ONLY REAL TEST
ENVIRONMENT
when thousands of things happen every second,
new, weird and unforeseen things happen all the time,
no test can anticipate everything
(but testing is good anyway, just don’t think you got everything covered)

55. KTHXBAI
@iconara
github.com/iconara
architecturalatrocities.com
burtcorp.com

56. COME TO SWEDEN
IN MARCH AND
TALK ABOUT BIG DATA
scandevconf.se/2013/call-for-proposals

57. IDEMPOTENCE

58. f(f(x)) = f(x)
doing something again doesn’t change the outcome

59. IDEMPOTENCE
if you don’t have to worry about things accidentally
happening twice, everything becomes much simpler

60. COUNTING UNIQUES
when adding to a set it doesn’t matter how many
times you do it, the end result is the same

61. INC X VS SET X
increments are not idempotent, and very scary,
if you can avoid non-idempotent operations, try

62. KTHXBAI
@iconara
github.com/iconara
architecturalatrocities.com
burtcorp.com