Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1DKnwXr. Peter Bourgon provides a practical introduction to Conflict-free Replicated Data Types (CRDTs) and describes a production CRDT system built at SoundCloud to serve several product features. Filmed at qconsf.com. Peter Bourgon is a distributed systems engineer who has seen things. He works at SoundCloud, building and improving the infrastructure that powers the world's largest audio platform.

1. Consistency without consensus:
CRDTs in production at SoundCloud

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/crdt-soundcloud

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Consistency without consensus:
CRDTs in production at SoundCloud

5. Me
Some guy
Embedded, sensor networks
Distributed systems
SoundCloud infrastructure

6. Theory

7. Distributed programming
“The art of solving the same problem that
you can solve on a single computer
using multiple computers.”
—book.mixu.net

8. Distributed programming
“Generally a bad idea,
best avoided.”
—me

9. >>> x = 1
>>> print x
1

10. $ curl -XPOST -d'{"val": 1}' http://db/vars/x
HTTP 502 Bad Gateway
$ curl -XGET http://db/vars/x
HTTP 503 Service Unavailable

11. Idioms

12. 1980s — RPC

13. 1990s — CORBA

14. 2000 — CAP

15. Partition tolerance
“The system continues to operate
despite message loss due to network
and/or node failure.”
—book.mixu.net

16. Partition
tolerance
Consistency Availability
✗
CP AP

17. CP
Chubby, Doozer — Paxos
ZooKeeper — Zab
Consul, etcd — Raft
? — Viewstamped Replication

18. AP
Cassandra
Riak
Mongo
Couch

19. Message failure
Delayed
Dropped
Delivered out-of-order
Duplicated

20. CALM principle
Consistency
As
Logical
Monotonicity

21. ACID 2.0
Associative
Commutative
Idempotent
Distributed, sure, whatever

22. CRDT
Conflict-free
Replicated
Data
Type

23. Increment-only
counter

24. A'
A → A'

25. +
(1 + 2) + 3 = 1 + (2 + 3)
1 + 2 = 2 + 1
1 + 1 ≠ 1

26. ∪
( 1 ∪ 2 ) ∪ 3 = 1 ∪ ( 2 ∪ 3 )
1 ∪ 2 = 2 ∪ 1
1 ∪ 1 = 1

27. { }
{ }
{ }

28. { }
{ }
{ }
123

29. { }
{ }
{ }
123
123

30. { }
{ }
{ }
123
123

31. { }
{ }
{ }
123
123
123

32. { }
{ }
{ }
123
123
123

33. { }
{ }
{ }
123
123
123
456

34. { }
{ }
{ }
123
123
123, 456
456

35. { }
{ }
{ }
123
123
123, 456
✘

36. { }
{ }
{ }
123
123, 456
123, 456

37. { }
{ }
{ }
123
123, 456
123, 456

38. Read
{123, 456} ∪ {123} ∪ {123, 456} = {123, 456}
{123, 456} ∆ {123} ∆ {123, 456} = {456}

39. { }
{ }
{ }
123
123, 456
123, 456
456

40. { }
{ }
{ }
123, 456
123, 456
123, 456

41. { }
{ }
{ }
123, 456
123, 456
123, 456

42. { }
{ }
{ }
123, 456
123, 456
123, 456

43. Interlude —
Bending the problem

44. CRDTs in production

46. Event
Timestamp
Actor
Verb
Thing

47. Event
2014-04-01T15:16:17.187Z
snoopdogg
reposted
theeconomist/election-day

48. Fan out on write
༼⍢༽ ●
●●●● (•ི̛ᴗ•̛)ྀ
●●● ༼•͟ ͜•༽
●●●●● (ಠ_ಠ)
●
●
●
●

49. Fan in on read
༼⍢༽ •••••
(•ི̛ᴗ•̛)ྀ
༼•͟ ͜•༽
(ಠ_ಠ)
[¬º-°] •••
••••••

50. Unique events — use a set
G-set — can’t delete
2P-set — add, remove once
OR-set — storage overhead

51. A wild set appears

52. Roshi set
S+ { A/1 B/2 C/3 }
S– { D/4 }
S { A B C }

53. Roshi set
S = actor’s outbox key
snoopdogg·outbox
A/B/C/D = actor+verb+thing
snoopdogg·repost·theeconomist/election-day
1/2/3 = timestamp
2014-04-01T15:16:17.187Z

54. Reading is easy

55. Writing is interesting

56. Insert
• If either key+ or key– already contains element,
and the existing score >= score,
no-op and exit.
• Insert (element, score) into add set key+.
• Delete (element) from remove set key–.

57. Delete
• If either key+ or key– already contains element,
and the existing score >= score,
no-op and exit.
• Insert (element, score) into add set key–.
• Delete (element) from remove set key+.

58. Example

59. S+ { A/1 B/2 }
S– { C/3 }

60. S+ { A/1 B/2 }
S– { C/3 }
Insert D/4

61. S+ { A/1 B/2 D/4 }
S– { C/3 }
Insert D/4

62. S+ { A/1 B/2 D/4 }
S– { C/3 }

63. S+ { A/1 B/2 D/4 }
S– { C/3 }
Insert D/4

64. S+ { A/1 B/2 D/4 }
S– { C/3 }
Insert D/4

65. S+ { A/1 B/2 D/4 }
S– { C/3 }

66. S+ { A/1 B/2 D/4 }
S– { C/3 }
Delete D/3

67. S+ { A/1 B/2 D/4 }
S– { C/3 }
Delete D/3

68. S+ { A/1 B/2 D/4 }
S– { C/3 }

69. S+ { A/1 B/2 D/4 }
S– { C/3 }
Delete D/5

70. S+ { A/1 B/2 D/4 }
S– { C/3 D/5 }
Delete D/5

71. S+ { A/1 B/2 }
S– { C/3 D/5 }

72. S+ { A/1 B/2 }
S– { C/3 D/5 }
Delete D/6

73. S+ { A/1 B/2 }
S– { C/3 D/6 }
Delete D/6

74. S+ { A/1 B/2 }
S– { C/3 D/6 }

75. Making it real

77. Pool
Cluster

78. Pool
Cluster
Pool
Cluster
Pool
Cluster
Farm

79. Writing is easy

80. Reading is interesting

81. Pool
Cluster
Pool
Cluster
Pool
Cluster
Farm

82. Pool
Cluster
Pool
Cluster
Pool
Cluster
Farm

83. ClusterCluster Cluster
{A B C} {A C} {A B C}
∪ = {A B C}
∆ = {B}

84. Pool
Cluster
Pool
Cluster
Pool
Cluster
Farm

85. github.com/soundcloud/roshi

86. In conclusion,

87. Consistency without consensus = CRDT.
Embrace your invariants.
Maybe bend your problem, not your solution.

88. Thanks!
soundcloud.com/jobs
@peterbourgon
☞ ☜

89. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/crdt-
soundcloud