The document discusses several anti-patterns when building distributed systems with Akka actors including: using global mutable state accessible by multiple actors; flat actor hierarchies without proper boundaries; having too many isolated actor systems instead of using custom dispatchers for bulkheading; logging in a way that negatively impacts performance; and not understanding hardware configurations can lead to suboptimal systems. The presentation provides alternatives such as immutable messaging, building hierarchies, using custom dispatchers, Akka's logging facilities, and hardware-aware configurations.
3. WHY THIS TALK?
Tools, of course, can be the subtlest of traps
— Neil Gaiman
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
4. MANUEL.BERNHARDT.IO
> Helping teams to get started with
reactive systems...
> ...and to keep them running
> Lightbend training partner (Akka,
Advanced Akka, Scala)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
5. MANUEL.BERNHARDT.IO
> Helping teams to get started with
reactive systems...
> ...and to keep them running
> Lightbend training partner (Akka,
Advanced Akka, Scala)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
6. If you want to make enemies, try
to change something.
— Woodrow Wilson
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
8. > it's okay for an actor to have mutable state
> as long as it retains total control over it and is the
only one to see it
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
9. HOW CAN IT HAPPEN?
> reference to mutable state in messages
> closing over mutable state in asynchronous calls
> passing shared mutable state to an actor's
constructor
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
10. WHY IS THIS BAD?
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
15. WHAT TO DO INSTEAD
> use immutable messages for state updates (e.g.
broadcasting for configuration changes)
> use queries for state inquiries (e.g. using the ask
pattern)
> for asynchronous calls, always use the pipe pattern
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
19. HOW CAN IT HAPPEN?
"Ceci n'est pas une hierarchie" - Magritte, 1928
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
20. IF YOUR ACTOR SYSTEM HAS NO HIERARCHY
YOU ARE MISSING THE POINT.
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
21. WHY IS THIS BAD?
> actor systems are designed to handle failures through
hierarchy
> ergo: no hierarchy, no failure handling
> why then bother to use actors in the first place?
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
22. How exception catch blocks are used in Java projects 1
1
Analysis of Exception Handling Patterns in Java Projects: An Empirical Study (S. Nakshatri, M. Hegde, S. Thandra)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
23. WHAT TO DO INSTEAD?
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
24. WHAT TO DO INSTEAD?
Build hierarchies.
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
26. The mother of excess is not joy
but joylessness.
— Friedrich Nietzsche
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
27. ANTI-PATTERN #3
TOO MANY ACTOR SYSTEMS
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
28. HOW CAN IT HAPPEN?
> eagerly wanting to isolate things
> and not understanding how Akka
works
> (but the intent is good)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
29. WHY IS THIS BAD?
> each actor system has at least one dispatcher backed
by a thread pool
> multiple actor systems = multiple thread pools,
contending for the same resources
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
30. WHAT TO DO INTEAD?
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
31. WHAT TO DO INSTEAD?
Bulkheading with custom dispatchers
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
36. HOW CAN THIS HAPPEN?
> string concatentation
log().debug("Received message: " + msg.toString());
> non-asynchronous logging
> not turning debug logging off in
production settings
> let's get real: logging to files (it is
2017)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
37. WHY IS THIS BAD?
> actor sytems are meant to process millions of
messages per second
> getting logging wrong has a huge performance impact
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
38. WHAT TO DO INSTEAD?
> use Akka's built-in logging facility
> carefully configure the logback appenders, use
asynchronous variants
> use string interpolation: log().debug("Received
message {}", msg);
> use a logging aggregation mechanism (logstash &
friends)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
40. All that we see or seem is but a
dream within a dream.
— Edgar Allan Poe
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
41. ANTI-PATTERN #5BEING OUT OF TOUCH WITH THE HARDWARE
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
42. HOW CAN THIS HAPPEN?
XKCDE 2
2
http://xkcd.com/1764/
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
43. WHY IS THIS A BAD THING?
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
44. WHY IS THIS A BAD THING?
fork-join-executor {
# Min number of threads to cap factor-based parallelism number to
parallelism-min = 2
# Parallelism (threads) ... ceil(available processors * factor)
parallelism-factor = 2.0
# Max number of threads to cap factor-based parallelism number to
parallelism-max = 10
}
ceil(available processors * factor)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
45. WHY IS THIS BAD?
> suboptimal or simply wrong configuration for the
amount of CPU cores
> costs of context switching
> contending for network
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
46. WHAT TO DO INSTEAD?
> know your hardware and configure accordingly
> beware of virtualization (is the hypervisor lying to
you?)
> run load tests on the same hardware as your target
production system
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
48. "...some men aren't looking for
anything logical, like money. They
can't be bought, bullied, reasoned,
or negotiated with. Some men just
want to watch the world burn."
— Alfred Pennyworth, Batman, The Dark Knight
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
50. HOW CAN THIS HAPPEN?
> calling a synchronous API
> or calling an API that calls an API that calls an API...
that calls a synchronous API
> explicitly waiting for the completion of a Future
> using Thread.sleep (don't!)
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
51. WHAT TO DO INSTEAD?
> if you really must (legacy API), use a dedicated
dispatcher optimized for the blocking case (and limited
in resources)
> always use Future in combination with the pipe pattern
> use the akka scheduler if you are waiting for something
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
52. ASK AND PIPE
def receive = {
case ComputePi(precision) =>
val originalSender = sender()
implicit val timeout = Timeout(5.seconds)
val computation: Future[Pi] = piActor ? Compute(precision, originalSender)
val result = computation.recover { case t: AskTimeoutException =>
ComputationTimeout(originalSender, precision)
}
result pipeTo self
case Pi(value, originalSender) =>
originalSender ! PiResult(value)
case ComputationTimeout(originalSender, precision) =>
originalSender ! ComputationFailed(s"Sorry, $precision was too high.")
}
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
54. A man who dares to waste one
hour of time has not discovered
the value of life.
— Charles Darwin
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
63. HOW CAN THIS HAPPEN?
> leaving the default on
> Java serialization over the wire
> Java serialization in Akka Persistance
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
64. WHY IS IT BAD?
> performance penalty!
> poor candidate for protocol evolution - message
evolutions result in older components not able to
process them any longer
> persistance - messages and snapshots can't be
processed any longer after changes
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
65. WHAT TO DO INSTEAD?
> use a proper binary format
> protobuf, avro, thrift
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu
66. THANK YOU
> Questions, comments, feedback?
> Contact me at manuel@bernhardt.io / @elmanu
> Check out more anti-patterns at https://
manuel.bernhardt.io
Scala Days 2017 Copenhagen - manuel.bernhardt.io - @elmanu