Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Akka in Production: Our Story

55 355 vues

Publié le

Everyone in the Scala world is using or looking into using Akka for low-latency, scalable, distributed or concurrent systems. We want to share our story of developing and productionizing multiple Akka apps, including low-latency ingestion and real-time processing systems, and Spark-based applications.

When does one use actors vs futures?
Why did we go with Logback instead of Akka's built-in logging?
Can we use Akka with, or in place of, Storm?
How did we set up instrumentation and monitoring in production?
How does one use VisualVM to debug Akka apps in production?
What happens if the mailbox gets full?
What is our Akka stack like?
We will share best practices that we've discovered when building Akka and Scala apps, pitfalls and things we'd like to avoid, and a vision of where we would like to go for ideal Akka monitoring, instrumentation, and debugging facilities.

  • Like to know how to take easy surveys and get huge checks - then you need to visit us now! Having so many paid surveys available to you all the time let you live the kind of life you want. learn more...■■■ https://tinyurl.com/realmoneystreams2019
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • I went from getting $3 surveys to $500 surveys every day!! learn more... ■■■ https://tinyurl.com/make2793amonth
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • I'll tell you right now (and I've got proof), that anyone who tells you "size doesn't matter to women" is flat out lying to your face and trying to make you feel better... Heck, just recently I asked a focus group of women via an anonymous online survey if size matters, and again and again they said "Oh my god, I HATE IT when it's SMALL." For a long time I didn't know what to tell the guys who'd write in to me and ask how to get "bigger." I'd say something lame like "Women actually like guys who are smaller... you just have to get good with your hands." Then I found "THE BIBLE of Penis Enlargement" by this guy named John Collins ●●● http://ishbv.com/pebible/pdf
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Like to know how to take easy surveys and get huge checks - then you need to visit us now! Having so many paid surveys available to you all the time let you live the kind of life you want. learn more...♣♣♣ http://ishbv.com/surveys6/pdf
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Akka in Production: Our Story

  1. 1. Akka in Production Our Story Evan Chan PNWScala 2013 Saturday, October 19, 13
  2. 2. Who is this guy? • Staff Engineer, Compute and Data Services, Ooyala • Building multiple web-scale real-time systems on top of C*, Kafka, Storm, etc. • github.com/velvia • Author of ScalaStorm, Scala DSL for Storm • @evanfchan 2 Saturday, October 19, 13
  3. 3. WANT REACTIVE? event-driven, scalable, resilient and responsive 3 Saturday, October 19, 13
  4. 4. SCALA AND AKKA AT OOYALA 4 Saturday, October 19, 13
  5. 5. COMPANY OVERVIEW Founded in 2007 Commercially launch in 2009 230+ employees in Silicon Valley, LA, NYC, London, Paris, Tokyo, Sydney & Guadalajara Global footprint, 200M unique users, 110+ countries, and more than 6,000 websites Over 1 billion videos played per month and 2 billion analytic events per day 25% of U.S. online viewers watch video powered by Ooyala CONFIDENTIAL—DO NOT DISTRIBUTE Saturday, October 19, 13 5
  6. 6. How we started using Scala • Ooyala was a mostly Ruby company - even MR jobs • Lesson - don’t use Ruby for big data • Started exploring Scala for real-time analytics and MR • Realized a 1-2 orders of magnitude performance boost from Scala • Today use Scala, Akka with Storm, Spark, MR, Cassandra, all new big data pipelines Saturday, October 19, 13
  7. 7. Ingesting 2 Billion Events / Day Consumer watches video Storm Nginx Raw Log Feeder Kafka New Stuff Saturday, October 19, 13
  8. 8. Livelogsd - Akka/Kafka file tailer Current File Coordinator Rotated File File Reader Actor File Reader Actor Kafka Kafka Feeder Saturday, October 19, 13 Rotated File 2
  9. 9. Storm - with or without Akka? Kafka Spout • Actors talking to each other within a bolt for locality • Don’t really need Actors in Storm Bolt • In production, found Storm too complex to troubleshoot Actor Actor Saturday, October 19, 13 • It’s 2am - what should I restart? Supervisor? Nimbus? ZK?
  10. 10. Akka Cluster-based Pipeline Kafka Consumer Kafka Consumer Kafka Consumer Kafka Consumer Kafka Consumer Spray endpoint Spray endpoint Spray endpoint Spray endpoint Spray endpoint Cluster Router Cluster Router Cluster Router Cluster Router Cluster Router Processing Actors Processing Actors Processing Actors Processing Actors Processing Actors Saturday, October 19, 13
  11. 11. Lessons Learned • Still too complex -- would we want to get paged for this system? • Akka cluster in 2.1 was not ready for production (newer 2.2.x version is stable) • Mixture of actors and futures for HTTP requests became hard to grok • Actors were much easier for most developers to understand Saturday, October 19, 13
  12. 12. Simplified Ingestion Pipeline Kafka Partition 1 Kafka SimpleConsumer Kafka Partition 2 Kafka SimpleConsumer • Kafka used to partition messages • Single process - super simple! • No distribution of data Converter Actor Converter Actor Cassandra Writer Actor Cassandra Writer Actor Saturday, October 19, 13 • Linear actor pipeline very easy to understand
  13. 13. STACKABLE ACTOR TRAITS 13 Saturday, October 19, 13
  14. 14. Why Stackable Traits? • Keep adding monitoring, logging, metrics, tracing code gets pretty ugly and repetitive • We want some standard behavior around actors -- but we need to wrap the actor Receive block: class someActor extends Actor { def wrappedReceive: Receive = { case x => blah } def receive = { case x => println(“Do something before...”) wrappedReceive(x) println(“Do something after...”) } } Saturday, October 19, 13
  15. 15. Start with a base trait... trait /** * */ def ActorStack extends Actor { Actor classes should implement this partialFunction for standard actor message handling wrappedReceive: Receive /** Stackable traits should override and call super.receive(x) for * stacking functionality */ def receive: Receive = { case x => if (wrappedReceive.isDefinedAt(x)) wrappedReceive(x) else unhandled(x) } } Saturday, October 19, 13
  16. 16. Instrumenting Traits... trait Instrument1 extends ActorStack { override def receive: Receive = { case x => println("Do something before...") super.receive(x) println("Do something after...") } } trait Instrument2 extends ActorStack { override def receive: Receive = { case x => println("Antes...") super.receive(x) println("Despues...") } } Saturday, October 19, 13
  17. 17. Now just mix the Traits in.... • Traits add instrumentation; Actors stay clean! • Order of mixing in traits matter class DummyActor extends Actor with Instrument1 with Instrument2 { def wrappedReceive = { case "something" => println("Got something") case x => println("Got something else: " + x) } } Antes... Do something before... Got something Do something after... Despues... Saturday, October 19, 13
  18. 18. PRODUCTIONIZING AKKA 18 Saturday, October 19, 13
  19. 19. Our Akka Stack • Spray - high performance HTTP • SLF4J / Logback • Yammer Metrics • spray-json • Akka 2.x • Scala 2.9 / 2.10 Saturday, October 19, 13
  20. 20. On distributed systems: “The only thing that matters is Visibility” 20 Saturday, October 19, 13
  21. 21. Using Logback with Akka • Pretty easy setup • Include the Logback jar • In your application.conf: event-handlers = ["akka.event.slf4j.Slf4jEventHandler"] • Use a custom logging trait, not ActorLogging • ActorLogging does not allow adjustable logging levels • Want the Actor path in your messages? • Saturday, October 19, 13 org.slf4j.MDC.put(“actorPath”, self.path.toString)
  22. 22. Using Logback with Akka trait Slf4jLogging extends Actor with ActorStack { val logger = LoggerFactory.getLogger(getClass) private[this] val myPath = self.path.toString logger.info("Starting actor " + getClass.getName) override def receive: Receive = { case x => org.slf4j.MDC.put("akkaSource", myPath) super.receive(x) } } Saturday, October 19, 13
  23. 23. Akka Performance Metrics • We define a trait that adds two metrics for every actor: • frequency of messages handled (1min, 5min, 15min moving averages) • time spent in receive block • All metrics exposed via a Spray route /metricz • Daemon polls /metricz and sends to metrics service • Would like: mailbox size, but this is hard Saturday, October 19, 13
  24. 24. Akka Performance Metrics trait ActorMetrics extends ActorStack { // Timer includes a histogram of wrappedReceive() duration as well as moving avg of rate of invocation val metricReceiveTimer = Metrics.newTimer(getClass, "message-handler", TimeUnit.MILLISECONDS, TimeUnit.SECONDS) override def receive: Receive = { case x => val context = metricReceiveTimer.time() try { super.receive(x) } finally { context.stop() } } } Saturday, October 19, 13
  25. 25. Performance Metrics (cont’d) Saturday, October 19, 13
  26. 26. Performance Metrics (cont’d) Saturday, October 19, 13
  27. 27. Flow control • By default, actor mailboxes are unbounded • Using bounded mailboxes • When mailbox is full, messages go to DeadLetters • mailbox-push-timeout-time: how long to wait when mailbox is full • Doesn’t work for distributed Akka systems! • Real flow control: pull, push with acks, etc. • Works anywhere, but more work Saturday, October 19, 13
  28. 28. Flow control (Cont’d) • A working flow control system causes the rate of all actor components to be in sync. • Witness this message flow rate graph of the start of event processing: Saturday, October 19, 13
  29. 29. VisualVM and Akka • Bounded mailboxes = time spent enqueueing msgs Saturday, October 19, 13
  30. 30. VisualVM and Akka • My dream: a VisualVM plugin to visualize Actor utilization across threads Saturday, October 19, 13
  31. 31. Tracing Akka Message Flows • Stack trace is very useful for traditional apps, but for Akka apps, you get this: at akka.dispatch.Future$$anon$3.liftedTree1$1(Future.scala:195) ~[akka-actor-2.0.5.jar:2.0.5] at akka.dispatch.Future$$anon$3.run(Future.scala:194) ~[akka-actor-2.0.5.jar:2.0.5] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94) [akka-actor-2.0.5.jar:2.0.5] at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381) [akka-actor-2.0.5.jar:2.0.5] at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259) [akka-actor-2.0.5.jar:2.0.5] at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) [akka-actor-2.0.5.jar:2.0.5] at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479) [akka-actor-2.0.5.jar:2.0.5] at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [akka-actor-2.0.5.jar:2.0.5] • What if you could get an Akka message trace? --> trAKKAr message trace <-akka://Ingest/user/Super --> akka://Ingest/user/K1: Initialize akka://Ingest/user/K1 --> akka://Ingest/user/Converter: Data Saturday, October 19, 13
  32. 32. Tracing Akka Message Flows Saturday, October 19, 13
  33. 33. Tracing Akka Message Flows trait TrakkarExtractor extends TrakkarBase with ActorStack { import TrakkarUtils._ val messageIdExtractor: MessageIdExtractor = randomExtractor override def receive: Receive = { case x => lastMsgId = (messageIdExtractor orElse randomExtractor)(x) Collector.sendEdge(sender, self, lastMsgId, x) super.receive(x) } } • Trait sends an Edge(source, dest, messageInfo) to a local Collector actor • Aggregate edges across nodes, graph and profit! Saturday, October 19, 13
  34. 34. Good Akka development practices • Don't put things that can fail into Actor constructor • Default supervision strategy stops an Actor which cannot initialize itself • Instead use an Initialize message • Put your messages in the Actor’s companion object • Namespacing is nice Saturday, October 19, 13
  35. 35. PUTTING IT ALL TOGETHER 35 Saturday, October 19, 13
  36. 36. Akka Visibility, Minimal Footprint trait InstrumentedActor extends Slf4jLogging with ActorMetrics with TrakkarExtractor object MyWorkerActor { case object Initialize case class DoSomeWork(desc: String) } class MyWorkerActor extends InstrumentedActor { def wrappedReceive = { case Initialize => case DoSomeWork(desc) => } } Saturday, October 19, 13
  37. 37. Next Steps • Name? • Open source? • Talk to me if you’re interested in contributing Saturday, October 19, 13
  38. 38. THANK YOU And YES, We’re HIRING!! ooyala.com/careers Saturday, October 19, 13