Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Eventually-Consistent Data   Structures      Sean Cribbs   @seancribbs #CRDT   StrangeLoop 2012
I work for Basho       We make
Riak is     Eventually     ConsistentSo are Voldemort and Cassandra
No ACID!
Duals or Duels?
Duals or Duels?object-oriented / functional
Duals or Duels?object-oriented / functional     static / dynamic
Duals or Duels?object-oriented / functional     static / dynamic consistency / availability
Duals or Duels?object-oriented / functional     static / dynamic consistency / availability   throughput / latency
Duals or Duels?object-oriented / functional     static / dynamic consistency / availability   throughput / latency    thre...
Duals or Duels?object-oriented / functional     static / dynamic consistency / availability   throughput / latency    thre...
Safety / LivenessProving the Correctness of Multiprocess Programs - Leslie                 Lamport (March 1977)
Safety / LivenessProving the Correctness of Multiprocess Programs - Leslie                 Lamport (March 1977)     •Safet...
Safety / LivenessProving the Correctness of Multiprocess Programs - Leslie                 Lamport (March 1977)     •Safet...
Safety / Liveness Proving the Correctness of Multiprocess Programs - Leslie                  Lamport (March 1977)       •S...
EventualConsistency          Replicated      Loose coordination  3     Convergence
Eventual is Good   ✔ Fault-tolerant   ✔ Highly available   ✔ Low-latency
Consistency?           No clear winner!           Throw one out?       3             Keep both?B
Consistency?              No clear winner!              Throw one out?       3                  Keep both?B     Cassandra
Consistency?              No clear winner!              Throw one out?       3                  Keep both?B     Cassandra ...
Conflicts! A!   B!
Semantic        Resolution• Your app knows the domain - use  business rules to resolve• Amazon Dynamo’s shopping cart
Semantic           Resolution  • Your app knows the domain - use     business rules to resolve  • Amazon Dynamo’s shopping...
Conflict-Free Replicated Data Types
Conflict-Free Replicated Data Types       useful abstractions
Conflict-Free          Replicated          Data Types     multipleindependent copies   useful abstractions
resolves automatically                      toward a single value         Conflict-Free          Replicated          Data T...
http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf
Bounded Join Semi-Lattices
Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉
Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set
Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on   S
Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on   S ‣ ⊥∈S
Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on   S ‣ ⊥∈S ‣ ∀x, y ∈ S: x ≤S y...
Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on   S ‣ ⊥∈S ‣ ∀x, y ∈ S: x ≤S y...
lmax LatticeS≔ℛa ⊔b ≔ max(a,b)⊥ ≔ -∞
lset Lattice                             {a,b,c,d,e}                 {a,b,c,d}                 {b,c,d,e}                  ...
CRDT Flavors• Convergent: State • Weak messaging requirements•Commutative: Operations •Reliable broadcast required •Causal...
Convergent CRDTs
Commutative  CRDTs
RegistersA place to put your stuff
Registers• Last-Write Wins (LWW-Register) • e.g. Columns in Cassandra• Multi-Valued (MV-Register) • e.g. Objects (values) ...
Counters Keeping tabs
G-Counter
G-Counter// Starts empty[]
G-Counter// Starts empty[]// A increments twice, forwarding state[{a,1}] // == 1[{a,2}] // == 2
G-Counter// Starts empty[]// A increments twice, forwarding state[{a,1}] // == 1[{a,2}] // == 2             // B increment...
G-Counter// Starts empty[]// A increments twice, forwarding state[{a,1}] // == 1[{a,2}] // == 2                 // B incre...
PN-Counter// A PN-Counter{  P = [{a,10},{b,2}],  N = [{a,1},{c,5}]}// == (10+2)-(1+5) == 12-6 == 6
SetsMembers Only
G-Set
G-Set// Starts empty{}
G-Set// Starts empty{}// A adds a and b, forwarding state{a}{a,b}
G-Set// Starts empty{}// A adds a and b, forwarding state{a}{a,b}             // B adds c             {c}
G-Set// Starts empty{}// A adds a and b, forwarding state{a}{a,b}             // B adds c             {c}// Merging{a,b,c}...
2P-Set
2P-Set// Starts empty{A={},R={}}
2P-Set// Starts empty{A={},R={}}// A adds a and b, forwarding state,// removes a{A={a}, R={}}   // == {a}{A={a,b},R={}} //...
2P-Set// Starts empty{A={},R={}}// A adds a and b, forwarding state,// removes a{A={a}, R={}}   // == {a}{A={a,b},R={}} //...
2P-Set// Starts empty{A={},R={}}// A adds a and b, forwarding state,// removes a{A={a}, R={}}   // == {a}{A={a,b},R={}} //...
LWW-Element-Set
OR-Set
G = (V,E)Graphs   E⊆V×V
G = (V,E)Graphs   E⊆V×V
G = (V,E)Graphs   E⊆V×V
Use-Cases• Social graph (OR-Set or a Graph)• Web page visits (G-Counter)• Shopping Cart (Modified OR-Set)• “Like” button (U...
Challenges: GC• CRDTs are inefficient• Synchronization may be required
Challenges:    Responsibility• Client • Erlang: mochi/statebox • Clojure: reiddraper/knockbox • Ruby: aphyr/meangirls, bke...
Thanks
Eventually Consistent Data Structures (from strangeloop12)
Prochain SlideShare
Chargement dans…5
×

Eventually Consistent Data Structures (from strangeloop12)

There are many reasons to use an eventually-consistent database — like Riak, Voldemort, or Cassandra — including increased availability, lower latency, and fault-tolerance. However, doing so requires a mental shift in how to structure client applications, and certain types of traditional data-structures, like sets, registers, and counters can’t be resolved simply in the face of race-conditions. It is difficult to achieve “logical monotonicity” except for the most trivial data-types.

That is, until the advent of Convergent Replicated Data Types (CRDTs). CRDTs are data-structures that tolerate eventual consistency. They replace traditional data-structure implementations and all have the property that, given any number of conflicting versions of the same datum, there is a single state on which they converge (monotonicity). This talk will discuss some of the most useful CRDTs and how to apply them to solve real-world data problems.

  • Soyez le premier à commenter

Eventually Consistent Data Structures (from strangeloop12)

  1. Eventually-Consistent Data Structures Sean Cribbs @seancribbs #CRDT StrangeLoop 2012
  2. I work for Basho We make
  3. Riak is Eventually ConsistentSo are Voldemort and Cassandra
  4. No ACID!
  5. Duals or Duels?
  6. Duals or Duels?object-oriented / functional
  7. Duals or Duels?object-oriented / functional static / dynamic
  8. Duals or Duels?object-oriented / functional static / dynamic consistency / availability
  9. Duals or Duels?object-oriented / functional static / dynamic consistency / availability throughput / latency
  10. Duals or Duels?object-oriented / functional static / dynamic consistency / availability throughput / latency threaded / evented
  11. Duals or Duels?object-oriented / functional static / dynamic consistency / availability throughput / latency threaded / evented safety / liveness
  12. Safety / LivenessProving the Correctness of Multiprocess Programs - Leslie Lamport (March 1977)
  13. Safety / LivenessProving the Correctness of Multiprocess Programs - Leslie Lamport (March 1977) •Safety: “nothing bad happens” (partial correctness)
  14. Safety / LivenessProving the Correctness of Multiprocess Programs - Leslie Lamport (March 1977) •Safety: “nothing bad happens” (partial correctness) •Liveness: “something good eventually happens” (termination)
  15. Safety / Liveness Proving the Correctness of Multiprocess Programs - Leslie Lamport (March 1977) •Safety: “nothing bad happens” (partial correctness) •Liveness: “something good eventually happens” (termination)“Safety and liveness: Eventual consistency is not safe” - Peter Bailishttp://www.bailis.org/blog/safety-and-liveness-eventual-consistency-is- not-safe/
  16. EventualConsistency Replicated Loose coordination 3 Convergence
  17. Eventual is Good ✔ Fault-tolerant ✔ Highly available ✔ Low-latency
  18. Consistency? No clear winner! Throw one out? 3 Keep both?B
  19. Consistency? No clear winner! Throw one out? 3 Keep both?B Cassandra
  20. Consistency? No clear winner! Throw one out? 3 Keep both?B Cassandra Riak & Voldemort
  21. Conflicts! A! B!
  22. Semantic Resolution• Your app knows the domain - use business rules to resolve• Amazon Dynamo’s shopping cart
  23. Semantic Resolution • Your app knows the domain - use business rules to resolve • Amazon Dynamo’s shopping cart“Ad hoc approaches have proven brittle and error-prone”
  24. Conflict-Free Replicated Data Types
  25. Conflict-Free Replicated Data Types useful abstractions
  26. Conflict-Free Replicated Data Types multipleindependent copies useful abstractions
  27. resolves automatically toward a single value Conflict-Free Replicated Data Types multipleindependent copies useful abstractions
  28. http://db.cs.berkeley.edu/papers/UCB-lattice-tr.pdf
  29. Bounded Join Semi-Lattices
  30. Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉
  31. Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set
  32. Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on S
  33. Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on S ‣ ⊥∈S
  34. Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on S ‣ ⊥∈S ‣ ∀x, y ∈ S: x ≤S y x⊔y=y
  35. Bounded Join Semi-Lattices 〈S, ⊔, ⊥〉 ‣ S is a set ‣ ⊔ is a least-upper bound (join/merge) on S ‣ ⊥∈S ‣ ∀x, y ∈ S: x ≤S y x⊔y=y ‣ ∀x ∈ S: x ⊔ ⊥ = x
  36. lmax LatticeS≔ℛa ⊔b ≔ max(a,b)⊥ ≔ -∞
  37. lset Lattice {a,b,c,d,e} {a,b,c,d} {b,c,d,e} {b,c,d}Time {a,b,c} {c,d,e} {b,c} {c,d} {d,e} {a,b} {a} {b} {c} {d} {e}
  38. CRDT Flavors• Convergent: State • Weak messaging requirements•Commutative: Operations •Reliable broadcast required •Causal ordering sufficient
  39. Convergent CRDTs
  40. Commutative CRDTs
  41. RegistersA place to put your stuff
  42. Registers• Last-Write Wins (LWW-Register) • e.g. Columns in Cassandra• Multi-Valued (MV-Register) • e.g. Objects (values) in Riak
  43. Counters Keeping tabs
  44. G-Counter
  45. G-Counter// Starts empty[]
  46. G-Counter// Starts empty[]// A increments twice, forwarding state[{a,1}] // == 1[{a,2}] // == 2
  47. G-Counter// Starts empty[]// A increments twice, forwarding state[{a,1}] // == 1[{a,2}] // == 2 // B increments [{b,1}] // == 1
  48. G-Counter// Starts empty[]// A increments twice, forwarding state[{a,1}] // == 1[{a,2}] // == 2 // B increments [{b,1}] // == 1// Merging[{a,2}, {b,1}] [{a,1}, {b,1}]
  49. PN-Counter// A PN-Counter{ P = [{a,10},{b,2}], N = [{a,1},{c,5}]}// == (10+2)-(1+5) == 12-6 == 6
  50. SetsMembers Only
  51. G-Set
  52. G-Set// Starts empty{}
  53. G-Set// Starts empty{}// A adds a and b, forwarding state{a}{a,b}
  54. G-Set// Starts empty{}// A adds a and b, forwarding state{a}{a,b} // B adds c {c}
  55. G-Set// Starts empty{}// A adds a and b, forwarding state{a}{a,b} // B adds c {c}// Merging{a,b,c} {a,c}
  56. 2P-Set
  57. 2P-Set// Starts empty{A={},R={}}
  58. 2P-Set// Starts empty{A={},R={}}// A adds a and b, forwarding state,// removes a{A={a}, R={}} // == {a}{A={a,b},R={}} // == {a,b}{A={a,b},R={a}} // == {b}
  59. 2P-Set// Starts empty{A={},R={}}// A adds a and b, forwarding state,// removes a{A={a}, R={}} // == {a}{A={a,b},R={}} // == {a,b}{A={a,b},R={a}} // == {b} // B adds c {A={c},R={}} // == {c}
  60. 2P-Set// Starts empty{A={},R={}}// A adds a and b, forwarding state,// removes a{A={a}, R={}} // == {a}{A={a,b},R={}} // == {a,b}{A={a,b},R={a}} // == {b} // B adds c {A={c},R={}} // == {c}// Merging{A={a,b,c},R={a}} {A={a,c}, R={}}
  61. LWW-Element-Set
  62. OR-Set
  63. G = (V,E)Graphs E⊆V×V
  64. G = (V,E)Graphs E⊆V×V
  65. G = (V,E)Graphs E⊆V×V
  66. Use-Cases• Social graph (OR-Set or a Graph)• Web page visits (G-Counter)• Shopping Cart (Modified OR-Set)• “Like” button (U-Set)
  67. Challenges: GC• CRDTs are inefficient• Synchronization may be required
  68. Challenges: Responsibility• Client • Erlang: mochi/statebox • Clojure: reiddraper/knockbox • Ruby: aphyr/meangirls, bkerley/ hanover• Server
  69. Thanks

×