In this webinar by Lightbend’s Eric Loots, Scala & Tooling Practice Lead, and Kikia Carter, Principal Enterprise Architect, we use a simple yet powerful visualization of a 5-node, Raspberry Pi-based cluster to reveal the inner workings of Akka Cluster. In a matter of minutes, you will gain a strong understanding of clustering, even if you don’t know anything about Akka.
4. Akka Cluster
• Peer-to-peer based
• Nodes disseminate cluster state via gossipping
• Failure detection via phi accrual failure detector
• Every partition has a leader:
• Dynamic role
• Decides on key node state transitions
• Akka Cluster Singleton
• Component which is running on one and only one node at a time
5. Akka Cluster State Diagram
joining
up
leaving
exiting
removed
join
(leader action)
leaving
(leaderaction)
(leaderaction)
(fd)
(fd)
(fd)
Unreachable
down
(fd)
6. Afraid of Clustering too?
• Documentation is a challenge…
• Browsing the logs & correlating events is another challenge…
10. Indistinguishable failure scenarios
node-1 sees node-2 as unreachable
node-2
OS
JVM
App
node-1
OS
JVM
App
✗
Network
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
Node HW
failure
11. Indistinguishable failure scenarios
node-1 sees node-2 as unreachable
node-2
OS
JVM
App
node-1
OS
JVM
App
✗
Network
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
Node HW
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
JVM
Crash
12. Indistinguishable failure scenarios
node-1 sees node-2 as unreachable
node-2
OS
JVM
App
node-1
OS
JVM
App
✗
Network
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
Node HW
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
JVM
Crash
node-2
OS
JVM
App
node-1
OS
JVM
App
App
Crash
13. Indistinguishable failure scenarios
node-1 sees node-2 as unreachable
node-2
OS
JVM
App
node-1
OS
JVM
App
✗
Network
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
Node HW
failure
node-2
OS
JVM
App
node-1
OS
JVM
App
JVM
Crash
node-2
OS
JVM
App
node-1
OS
JVM
App
App
Crash
node-2
OS
JVM
App
node-1
OS
JVM
App
Long JVM
GC pauses
23. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
24. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
Subscribe to
Cluster Events
Subscribe to
Cluster Events
25. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
leader moves
node-0 to Up
Gossipping
26. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
MemberUp(node-0)
27. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
28. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
MemberUp(node-0
29. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
30. Cluster State visualisation
Behind the screens
Cluster
Status
Tracker
Actor
node-4
Akka Cluster Subsystem
Cluster
Status
Tracker
Actor
node-0
Akka Cluster Subsystem
. . .
Gossipping
31. Demo time!
• Let’s run through a couple of scenarios
• Basic cluster formation & reachability
• Split-brain caused by a network partition
• Manual recovery after inspection (Akka HTTP Management)
• Automatic recovery using Akka Cluster Split-Brain Resolver
33. Wrap-up
• Source code - Watch this repo for updates!
https://github.com/lightbend/Pi-Akka-Cluster
• We ran a 2-day Workshop on this topic at
• ScalaDays New York
• Reactive Summit Montreal
• Custom designed LED strip coming up — no need to solder
anymore!