Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Distributed in Space and Time

1 817 vues

Publié le

How do I create an application that is always available, even in the face of catastrophic hardware failures? How do I make it cope with large fluctuations in the number of active users or their behavior? The answers to both questions have one thing in common: the need to spread the application out, to distribute it. In this presentation we take a deep dive into the architecture of reactive applications (see http://reactivemanifesto.org/), starting from cheap and efficient best-effort messaging and building delivery guarantees and persistence on top—with examples using Akka. Understanding these mechanisms and how they relate to each other will allow you to build systems that strike the right balance between partition tolerance, consistency, availability, implementation complexity and operational cost.

Publié dans : Logiciels
  • Soyez le premier à commenter

Distributed in Space and Time

  1. 1. Dr. Roland Kuhn Akka Tech Lead @rolandkuhn Distributed in Space and Time
  2. 2. The Future is Bright 2 source: http://nature.desktopnexus.com/wallpaper/58502/
  3. 3. The Future is Bright • Computing Resources: Effectively Unbounded • 18-core CPU packages, 1024 around the corner • on-demand provisioning in the cloud • Demand for Web-Scale Apps: Ravenous • everyone has the tools • successful ideas are bought for Giga-$ 3
  4. 4. How do I make a Large-Scale App? • figure out a service to provide to everyone • this will require resources accordingly: • storage • computation • communication • scalability implies no shared contention points • need to be elastic ➟ use the cloud 4
  5. 5. How do I make an Always-Available App? • must not have single point of failure • all functions and resources are replicated • protect against data loss • protect against service downtime • regional content delivery • need to distribute ➟ use the cloud 5
  6. 6. How do I make a Persistent App? • runtime replication usually not sufficient • system of record needed • one central database is not tenable • persist “locally” and replicate changes 6
  7. 7. How do I make a Persistent App? 7
  8. 8. How do I make a Persistent App? 8
  9. 9. How do I make a Persistent App? • runtime replication usually not sufficient • system of record needed • one central database is not tenable • persist “locally” and replicate changes • temporally distributed change records • spatially distributed change log • granularity can be one Aggregate Root (Event Sourcing) • extract business intelligence from history • scale & distribute persistence ➟ in the cloud 9
  10. 10. As Noted Earlier: The Future is Bright 10 source: http://nature.desktopnexus.com/wallpaper/58502/
  11. 11. … or is it? 11 Photograph: Patrick Cromerford, 2011
  12. 12. Distribution: a Double-Edged Sword • did that message make it over the network? • is that other machine running? • are our data centers in sync? • did we lose messages in that hardware crash? 12
  13. 13. The Essence of the Problem A distributed system is one whose parts can fail independently of each other. 13
  14. 14. Fighting Back! 14 source: www.hear2heal.com (Kokopelli Rain Dance)
  15. 15. Transactional Storage Consensus Message Delivery …
  16. 16. Transactional Storage • Serializability simplifies reasoning about behavior • … but is at odds with distribution (CAP theorem) • Eventual Consistency is often sufficient
 (PeterBailis:ProbabilisticallyBoundedStaleness) • Scalability can be regained by partitioning
 (PatHelland:LifebeyondDistributedTransactions) • consistency has a cost • you need to choose how much is necessary 16
  17. 17. PAXOS and Friends • problem: distributed consensus • hard solutions favor Consistency • 2PC, PAXOS, RAFT • if a node is unreachable then consensus cannot be reached • soft solutions favor Availability • quorum, allow nodes to drop out • partitions can lead to more than one active set • consistency always has a cost • you need to choose which one is appropriate 17
  18. 18. Fighting Message Loss • resending is the only cure • sender stores the message and retries • recipient must send acknowledgement eventually • fighting unreliable medium is sender’s obligation • sender’s buffer must be persistent • resending must begin anew after a crash 18
  19. 19. At-Least-Once Delivery with Akka 19 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  20. 20. At-Least-Once Delivery with Akka 20 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  21. 21. At-Least-Once Delivery with Akka 21 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  22. 22. At-Least-Once Delivery with Akka 22 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  23. 23. At-Least-Once Delivery with Akka 23 class ALOD(other: ActorPath) extends PersistentActor with AtLeastOnceDelivery { val IDs = Iterator from 1 def receiveCommand = { case Command(x) => persist(ImportantResult(x)) { ir => sender() ! Done(x) deliver(other, FollowUpCommand(x, IDs.next(), _)) } case rc: ReceiptConfirmed => persist(rc) { rc => confirmDelivery(rc.deliveryId) } } def receiveRecover = { case ImportantResult(x) => deliver(other, FollowUpCommand(x, IDs.next(), _)) case ReceiptConfirmed(_, id) => confirmDelivery(id) } }
  24. 24. Mysterious Double-Bookings • acknowledgements can be lost • sender will faithfully duplicate the message • therefore named at-least-once • recipient is responsible for dealing with this 24
  25. 25. Exactly-Once Delivery • processing occurs only once for each message • all middleware does it* • EIP: Guaranteed Delivery just means persistence • JMS: configure session for AUTO_ACKNOWLEDGE • Google’s MillWheel 25
  26. 26. Exactly-Once Delivery with Akka 26 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  27. 27. Exactly-Once Delivery with Akka 27 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  28. 28. Exactly-Once Delivery with Akka 28 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  29. 29. Exactly-Once Delivery with Akka 29 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  30. 30. Exactly-Once Delivery with Akka 30 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { persist(rc) { rc => sender() ! rc doTheAction() nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  31. 31. Exactly-Once Delivery with Akka 31 class PersistThenAct extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { doTheAction() persist(rc) { rc => sender() ! rc nextId += 1 } } else if (id < nextId) sender() ! rc // otherwise an older command was lost, so wait for the resend } def receiveRecover = { case ReceiptConfirmed(id, _) => nextId = id + 1 } private def doTheAction(): Unit = () // actually do the action }
  32. 32. The Hard Truth CPU can stop between any two instructions, therefore exactly-once semantics cannot be Guaranteed*. *with a capital G, i.e. with 100% absolute coverage 32
  33. 33. Mitigating that Hard Truth • probability of loss or duplication can be small • many systems are fine with almost-exactly-once • still need to decide on which side to err • persist-then-act ➟ almost-exactly-but-at-most-once • act-then-persist ➟ almost-exactly-but-at-least-once 33
  34. 34. We Can Do Better! • an operation is idempotent if multiple application produces the same result as single application • (side-)effects must also be idempotent • unique transaction identifiers etc. • this achieves effectively-exactly-once • can save persistence round-trip latency by optimistic execution 34
  35. 35. Effectively-Exactly-Once with Akka 35 class Idempotent(other: ActorRef) extends PersistentActor { var nextId = 1 def receiveCommand = { case FollowUpCommand(x, id, deliveryId) => val rc = ReceiptConfirmed(id, deliveryId) if (id == nextId) { // optimistically execute if (isValid(x)) { updateState(x) other ! Command(x) // assuming idempotent receiver } else sender() ! Invalid(x) nextId += 1 // then take care of reliable delivery handling persistAsync(rc) { rc => sender() ! rc } } else if (id < nextId) sender() ! rc } ... }
  36. 36. The Future is Bright Again! 36 source: http://freewallpapersbackgrounds.com/
  37. 37. To Recapitulate: 37 Problem Choices Transactional Storage monolithic or distributed/partitioned, serializable or eventually consistent Cluster Management strong consensus or convergent gossip, consistency or availability Message Delivery at-most-once, at-least-once, almost-exactly-once, effectively-exactly-once … …
  38. 38. Conclusion: Consistency à la Carte • not all parts of a system have the same needs • expend the effort only where necessary • idempotency does not come for free • confirmation handling cannot always be abstracted away • incur the runtime overhead only where needed • persistence round-trips take time • consensus protocols take more time • include required semantics in system design 38
  39. 39. Remember that the trade-off is always between consistency and availability. And latency.
  40. 40. ©Typesafe 2014 – All Rights Reserved

×