The dark side of Akka and the remedy - bp.scala meetup

The dark side of Akka
and the remedy
Ákos Kriváchy

Introduction - Ákos Kriváchy
• Scala disciple/convert/fanatic since 2013 February
• First FP language
• Akka since 2014 February
• Things I love about Scala:
• Static typing
• Partial Functions
• foldLeft, tail recursion
2

Recap on Akka
• Akka in key points:
• Messaging
• Actors
• Mailboxes (ActorRefs)
• Hierarchy
• Supervision
• Location transparency
Source: http://www.scottlogic.com/blog/2014/08/15/using-akka-and-scala-to-render-a-mandelbrot-set.html
3

Example from previous Meetup: URL scraping
4

To Akka or not to Akka?
• Akka solves:
• Concurrency
• Scalability and distributability
• Resilience
• Akka in our team:
•We had to rewrite a legacy Java component
• Chose Akka to try it out (after 4 months of research)
5

Problem #1: Any and Actor Ref
• All messages are Any-s
• Anything that’s not
handled ends up as a “dead
letter”
• Essentially a loss of all
typesafety
• Requires extensive testing
to “feel safe”
class MyActor(databasePersistence: ActorRef,
emailSender: ActorRef,
MQSender: ActorRef,
widgetDao: ActorRef,
twitterService: ActorRef)
extends Actor {
def receive: Receive = {
case Message(data) =>
twitterService ! Tweet(data)
// ...
case OtherMessage(_) =>
// ...
}
}
}
6

Solution: Typed Channels
val channelA: ChannelRef[(MsgA, MsgB) :+: TNil] = ???
val a = new MsgA
channelA <-!- a // send a to channelA
a -!-> channelA // same thing as above
val fB: Future[MsgB] = channelA <-?- a // ask the actor
Instead of the current solution:
val fB: Future[Any] = actorRef ? a
http://doc.akka.io/docs/akka/2.2.0/scala/typed-channels.html 7

NOPE!
http://doc.akka.io/docs/akka/2.3.0-RC1/project/migration-guide-2.2.x-2.3.x.html
8

It’s on the roadmap
9
August 28, 2014: http://typesafe.com/blog/akka-roadmap-update-2014
Lastly, we dare a look beyond the next major release: after
Akka 2.4.0 [early 2015] we will tackle the number one feature
request, namely more type-safety for inter-Actor messaging.
This will be a major research effort and likely also a disruptive
API change. For this reason Akka 2.4 is planned to be a long-term
support release, because even though we will try to find
ways for compatibility or at least an easy migration path we
cannot currently foresee the exact nature of all related
changes.

In the meantime: Don’t use Actors for everything
• What do we use instead?
• Scala Futures are extremely powerful
• Futures compose more nicely than Actors
• When should we use Actors?
• State in the application
• A cache is not state
• Extreme scaling is needed (location transparency)
• Need the flexibility to place any operation on any JVM -> use Actors for everything
• Realtime data flow applications (not request/response)
• What about supervision and „let it crash”?
• Future’s handle failure also
10

Code: https://github.com/krivachy/AkkaWithFsm
11

class Getter(url: String, depth: Int) extends Actor with ActorLogging with LinkParser {
lazy val client = new NingWSClient(new Builder().build()).url(url)
override def preStart() = {
client.get().map(_.body).map(Response).pipeTo(self)
}
def receive = {
case Response(body) =>
val links = parseLinks(body).toList
log.info(s"URL $url at depth $depth had ${links.size} links.")
links.foreach {
link =>
log.info(s"Sending link '$link'")
context.parent ! Controller.Check(link, depth)
}
context.stop(self)
case Status.Failure(cause) =>
log.error(s"Failed to GET $url", cause)
context.stop(self)
}
}
12

Simplified with Futures:
object Getter extends LinkParser {
case class Urls(urls: Set[String], depth: Int)
lazy val client = new NingWSClient(new Builder().build())
def get(url: String, depth: Int)(implicit ec: ExecutionContext): Future[Urls] = {
client.url(url).get().map {
response =>
Urls(parseLinks(response.body).toSet, depth)
}
}
}
13

Problem #2: Hellish code complexity
14

var hell
• Actors have too much mutable state
• Our worst scenario: Actor with 300 lines and 20 vars
• Hard to reason about state when everything is “global” inside the
Actor
• How do you initialize state that is only used in some cases?
• var something: SomeType = _
• NPE
• var something: Option[SomeType] = None
• Always need to “getOrElse”
15

become/unbecome hell
• Pushing and popping state on a stack
• context.become(behavior: Receive, discardOld: Boolean = true)
• context.unbecome()
• “context.become” isn’t enforced to be called last
• You use methods to keep things short, but there will be multiple methods
trying to modify the behaviour
• i.e. you could end up inadvertently overwriting behavior
• One place: context.become(handleCoolMessages orElse waitForNewRequests)
• Somewhere else: context.become(waitForNewRequests, discardOld = true)
• When things blow up you have no idea how you got there
16

class Receptionist extends Actor {
def receive = waiting
def waiting: Receive = {
case Api.Scrape(url, depth) =>
context.become(next(Vector(Job(sender, url, depth))))
}
def running(queue: Vector[Job]): Receive = LoggingReceive {
case Result(links) =>
// …
context.become(next(queue.tail))
case Api.Scrape(url, depth) =>
context.become(enqueue(queue, Job(sender, url, depth)))
case Terminated(_) =>
// …
context.become(next(queue.tail))
}
def enqueue(queue: Vector[Job], job: Job): Receive = LoggingReceive {
// …
running(queue)
}
def next(queue: Vector[Job]): Receive = LoggingReceive {
// …
running(queue)
}
}
17

Debug hell
• Debugging Actors is hard
• Stacktraces lack meaning
• Need to have a lot of boilerplate utility code:
• What was the message?
• Who sent the message?
• What’s my internal state?
• Possible solution:
• always override def preRestart(reason: Throwable, message: Option[Any])
• Still don’t know how we ended up in that become/unbecome state.
18

[ERROR] [akka://simple-actor/user/receptionist/controller-1/$a] Exception happened
java.lang.Exception: exception happened here
at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)
at meetup.akka.simple.Getter$$anonfun$receive$1$$anonfun$1.apply(Getter.scala:37)
at scala.util.Try$.apply(Try.scala:191)
at meetup.akka.simple.Getter$$anonfun$receive$1.applyOrElse(Getter.scala:37)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at meetup.akka.simple.Getter.aroundReceive(Getter.scala:16)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
19

We need Logging for Debugging
• Akka provides:
• with ActorLogging
• log.info(…)
• LoggingRecieve: def receive = LoggingReceive { … }
• turn on: akka.actor.debug.receive=true
• Lifecycle logging: akka.actor.debug.lifecycle=true
• For all Actors
• Start, restart, stop, supervise, watch events
• Autorecieve logging: akka.actor.debug.autoreceive=true
• For all Actors
• Automatically handled messages: Stop, Kill, PoisionPill, Terminated, etc.
• Issue:
• Akka only provides logging of messages per Receive block (has pros and cons)
• If you missed one => good luck debugging issues around it in production
20

Logging example
[DEBUG] [akka://simple-actor/user/receptionist] started (meetup.akka.simple.Receptionist@14ba772)
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] started (meetup.akka.simple.Controller@38a5d7)
[DEBUG] [akka://simple-actor/user/receptionist] now supervising Actor[akka://simple-actor/user/receptionist/controller-1#4232237]
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] now watched by Actor[akka://simple-actor/user/receptionist#1565954732]
[DEBUG] [akka://simple-actor/user/receptionist/controller-1] received handled message
Check(http://doc.akka.io/docs/akka/2.3.5/intro/what-is-akka.html,1)
21

“Everything* is a Finite State Machine.”**
-me
*not everything
** do not quote me
23

FSM concepts
object Receptionist {
object Internal {
sealed trait State
case object Sleeping extends State
case object Processing extends State
sealed trait Data
case class NoQueue(requestId: Int = 0) extends Data
case class Queue(currentRequestId: Int, items: Vector[Job]) extends Data
}
}
class Receptionist extends FSM[Internal.State, Internal.Data] { … }
24

Define handlers for all states
// Initialize with data
startWith(Sleeping, NoQueue())
// Handlers for states
when(Sleeping)(enqueueNewRequest)
when(Processing) (processResult orElse enqueueNewRequest orElse reportError)
25

Enqueue New Request State Function
def enqueueNewRequest: StateFunction = {
case Event(Api.Scrape(url, depth), NoQueue(requestId)) =>
startControllerFor(requestId + 1, Vector(Job(sender(), url, depth)))
case Event(Api.Scrape(url, depth), queue: Queue) =>
if (queue.items.size > 3) {
stay replying Api.Failed(url)
} else {
goto(Processing) using
Queue(queue.currentRequestId, queue.items :+ Job(sender(), url, depth))
}
}
• Important:
• StateFunction: Event => State
• Event(incomingMessage, data) =>
• State transition: goto/stay (nextState) using (data) forMax(timeout) replying (message)
26

Monitoring state transitions
Internal:
onTransition {
case Idle -> Active => setTimer("timeout", Tick, 1 second, true)
case Active -> _ => cancelTimer("timeout")
case x -> Idle => log.info("entering Idle from " + x)
}
External:
monitoredActor ! SubscribeTransitionCallBack(self)
def recieve = {
case Transition(monitoredActor, oldState, newState) =>
if (newState == Errored) alert.raiseAlert(...)
}
override def postStop() = {
monitoredActor ! UnsubscribeTransitionCallBack(self)
}
27

Handling failure
whenUnhandled {
case Event(any, data) =>
val logUpToHere = prettyPrint(getLog)
log.error(s"Unhandled event: ${any}n$logUpToHere")
stay()
}
def failure: StateFunction = {
case Event(Status.Failure(cause), _) =>
log.error(s"Failed to GET $url", cause)
stop(FSM.Failure(cause))
}
onTermination {
case StopEvent(FSM.Normal, state, data) => ???
case StopEvent(FSM.Shutdown, state, data) => ???
case StopEvent(FSM.Failure(cause), state, data) => ???
} 28

Result is simplicity
class Receptionist extends FSM[Internal.State, Internal.Data] {
startWith(Sleeping, NoQueue())
when(Sleeping)(enqueueNewRequest)
when(Processing) (processResult orElse enqueueNewRequest orElse reportError)
def enqueueNewRequest: StateFunction = ???
def processResult: StateFunction = ???
def reportError: StateFunction = ???
private def nextQueueItem(queue: Queue): State = ???
private def startControllerFor(requestId: Int, queue: Vector[Job]): State = ???
whenUnhandled { ??? }
initialize()
}
29

Hell status
var ✔
No more mutable global state
inside Actors. Everything is
typed to the specific State.
become/unbecome ✔
All methods have to end in a
state transition. States are
clearly defined what they do.
debug/logging ?
30

LoggingFSM
• Remembers state transitions:
• override def logDepth = 8
• def getLog: Seq[LogEntry[Internal.State, Internal.Data]]
• Use with: onTermination
• Debug logging: akka.actor.debug.fsm=true
• Automatically logs important Events: message + internal data
• Logs state transitions
• Use with: akka.actor.debug.lifecycle=true
31

Debug log example
[DEBUG] [akka://fsm/user/receptionist] processing
Event(Result(Set([…])),Queue(1,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://doc.akka.io
/docs/akka/2.3.5/intro/what-is-akka.html,1)))) from Actor[akka://fsm/user/receptionist/controller-1#4232237]
[DEBUG] [akka://fsm/user/receptionist/controller-1] transition CollectingResults -> Completed
[DEBUG] [akka://fsm/user/receptionist] transition Processing -> Sleeping
[DEBUG] [akka://fsm/user/receptionist/controller-1] stopped
32

[ERROR] [akka://fsm/user/receptionist] Unhandled event: some string
Last 8 entries leading up to this point:
in state: Sleeping
with data: NoQueue(2)
received: Scrape(http://non-existent.link,5)
in state: Processing
with data: Queue(3,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://non-existent.
link,5)))
received: Result(Set(http://non-existent.link))
[…]
in state: Processing
with data:
Queue(4,Vector(Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent1,0),
Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent2,0),
Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent3,0),
Job(Actor[akka://fsm/system/testActor1#758674372],http://non.existent4,0)))
received: some string 33

Hell status
var ✔
No more mutable global state
inside Actors. Everything is
typed to the specific State.
become/unbecome ✔
All methods have to end in a
state transition. States are
clearly defined what they do.
debug/logging ✔
FSM does all the debug logging
we would ever need.
34

Conclusion: My experiences
Some people, when confronted with a scalability problem, think
“I know, I'll use Akka.” Now they have two problems.
• Akka is awesome, but:
• Needs a new mindset
• Unexperienced developers + learning curve + easy to make
mistakes = headaches
• In large teams (30+) where everyone is expected to be able to
change all code it becomes an issue.
• Use Actors only when really needed
• All Actors should probably be FSM
@krivachy https://github.com/krivachy/AkkaWithFsm

The dark side of Akka and the remedy - bp.scala meetup

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (10)

Similaire à The dark side of Akka and the remedy - bp.scala meetup

Similaire à The dark side of Akka and the remedy - bp.scala meetup (20)

Dernier

Dernier (20)

The dark side of Akka and the remedy - bp.scala meetup