Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2C0klwp. Tyler McMullen talks through the components and design of a real system, as well as the theory behind them. The system is built to perform very high volumes of health checks, done across a cluster of machines for reliability and scalability. It includes things like gossip, failure detection, leader election, logical clocks, and consistency trade-offs. Filmed at qconsf.com. Tyler McMullen is CTO at Fastly, where he’s responsible for the system architecture and leads the company’s technology vision. As part of the founding team, he built the first versions of Fastly’s Instant Purging system, API, and Real-time Analytics. Before Fastly, he worked on text analysis and recommendations at Scribd.

1. Building a
Skyscraper with
Legos:
The
Anatomy
of a Distributed
System

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
health-distributed-system

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Tyler McMullen
@tbmcmullen

6. ok, what’s up?

7. Let’s build a
distributed system!

8. The Project

20. 50+ Datacenters
Thousands of bare metal servers
up to 64 Servers per Datacenter
Ten of Thousands of Origin Servers

21. Recap

22. Still with me?

23. One
Possible
Solution

25. (Too) Strong Consistency

26. Eventual Consistency

27. Forward Progress

29. Ownership

31. Rendezvous Hashing

33. h(Sn,O) = Wn
hash function we decided upon
set of live servers
origin server we’re
deciding on the owner of
the weight or priority

34. priorities(O1) = [S2,
S3,
S1,
S4,
…]

35. priorities(O2) = [S4,
S2,
S3,
S1,
…]

36. Failure Detection

39. u up?
1 2
3
4
5
✅

40. 1 2
3
4
5
✅

41. 1 2
3
4
5
❌

42. Memberlist

43. Gossip

44. Push and Pull

45. Convergence

46. Causality

47. Burgers happened-before Calzone
Burgers happened-before Daal
Jack
Jill
Arugula
Burgers
Calzone
Daal
Arugula happened-before Burgers
Calzone is-concurrent-with Daal

48. Lattices

50. Causality

51. Version Vectors

52. S1
S2
S3
[1,0,0]
[1,0,0]
[0,0,1]
[1,1,0] [1,2,1][1,1,1]

53. [0,0,0]
[1,0,0] [0,1,0] [0,0,1]
[1,1,0]
[1,1,1]
[1,1,2][2,1,1]

54. Coordination-free
Distributed
Map

55. Version DeltaMerge

56. Δ-CRDT Map

57. type SharedMap struct {
storage map[Key]SharedMapRecord
v clock.VersionVector
}
type SharedMapRecord struct {
value Value
dot clock.VVDot
}

58. Send_Version:
Send our Version Vector.

59. Received_Version(V):
For each record(R) in our map:
If (V happened-before R.Dot) OR
(V is-concurrent-with R.Dot):
Add R to Delta.
Send Delta.

60. Received_Delta(D):
V = Our Version Vector
For each record(R) in D:
If R.Dot happened-before V:
Skip it.
R’ = Local Record
If R’.Dot happened-before D.Version:
Merge it.
R and R’ are concurrent: ✨

61. ✨ = Rendesvous

62. Delta-state
CRDT Map

64. Ok, why?

65. Edge Compute

66. Coordination-free
Distributed
Systems

67. Single System Image

68. We posit that striving for distributed
systems that provide “single system
image” semantics is fundamentally
flawed and at odds with how systems
operate in the physical world.

69. We need new
metaphors.

70. We need new
intuition.

71. Thank You.
@tbmcmullen

72. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
health-distributed-system