Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Akka streams
Streaming
Johan Andrén
Voxxeddays, Zürich, 2017-02-23
with
All the things!
Johan Andrén
Akka Team
Stockholm Scala User Group
@apnylle
johan.andren@lightbend.com
Make building powerful concurrent &
distributed applications simple.
Akka is a toolkit and runtime
for building highly con...
Actors – simple & high performance concurrency
Cluster / Remoting – location transparency, resilience
Cluster tools – and ...
Reactive Streams
Reactive Streams timeline
Oct 2013
RxJava, Akka and Twitter-
people meeting
“Soon thereafter” 2013
Reactive Streams
Expert...
Reactive Streams
Reactive Streams is an initiative to provide a
standard for asynchronous stream processing with
non-block...
Reactive Streams
Reactive Streams is an initiative to provide a
standard for asynchronous stream processing with
non-block...
Stream processing
Source Sink
Flow
Reactive Streams
Reactive Streams is an initiative to provide a
standard for asynchronous stream processing with
non-block...
Asynchronous stream processing
Source Sink
(possible)
asynchronous
boundaries
Flow
Reactive Streams
Reactive Streams is an initiative to provide a
standard for asynchronous stream processing with
non-block...
No back pressure
Source Sink
10 msg/s 1 msg/s
Flow
asynchronous
boundary
No back pressure
Source Sink
10 msg/s 1 msg/s
Flow
asynchronous
boundary
OutOfMemoryError!!
No back pressure - bounded buffer
Source Sink
10 msg/s 1 msg/s
Flow
buffer size 6
🗑
asynchronous
boundary
Async non blocking back pressure
Source Sink
1 msg/s
1 msg/s
Flow
buffer size 6
🗑
asynchronous
boundary
Hey! give me 2 more
Reactive Streams
RS Library A RS library B
async
boundary
Reactive Streams
“Make building powerful concurrent &
distributed applications simple.”
Complete and awesome
Java and Scala APIs
(Just like everything in Akka)
Akka Streams
Akka Streams in ~20 seconds:
final ActorSystem system = ActorSystem.create();

final Materializer materializer = ActorMate...
Akka Streams in ~20 seconds:
final ActorSystem system = ActorSystem.create();

final Materializer materializer = ActorMate...
Akka Streams in ~20 seconds:
implicit val system = ActorSystem()

implicit val mat = ActorMaterializer()



val source = S...
Akka Streams in ~20 seconds:
Source.range(0, 20000000)

.map(Object::toString)

.runForeach(str -> System.out.println(str)...
Akka Streams in ~20 seconds:
Source(0 to 20000000)

.map(_.toString)

.runForeach(println)
complete sources on github
Numbers as a service
final Source<ByteString, NotUsed> numbers = Source.unfold(0L, n -> {

long next = n + 1;

return Opti...
Numbers as a service
val numbers =

Source.unfold(0L) { (n) =>

val next = n + 1

Some((next, next))

}.map(n => ByteStrin...
recv buffer
send buffer
🚚
🚚
🚚
🚚
🚚
🚚
🚚
Back pressure over TCP numbers
TCP HTTP
Server
Client
recv buffer
send buffer
🚚
🚚
🚚
🚚
🚚
🚚
🚑
Back pressure over TCP numbers
TCP HTTP
Backpressure
Server
Client
recv buffer
send buffer
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚚
🚑
Back pressure over TCP numbers
TCP HTTP
Backpressure
Backpressure
Server
Client
A more useful example
complete sources on github
final Flow<Message, Message, NotUsed> measurementsFlow =

Flow.of(Message...
A more useful example
complete sources on github
val measurementsFlow =

Flow[Message].flatMapConcat(message =>

message.a...
The tale of the two pancake chefs
HungrySink
Frying
Pan
BatterSource
Scoops of batter
Pancakes
nom nom nom
asynchronous
bo...
Rolands pipelined pancakes
HungrySinkPan 2BatterSource Pan 1
nom nom nom
Rolands pipelined pancakes
Flow<ScoopOfBatter, HalfCookedPancake, NotUsed> fryingPan1 =

Flow.of(ScoopOfBatter.class).map(...
Rolands pipelined pancakes
// Takes a scoop of batter and creates a pancake with one side cooked
val fryingPan1: Flow[Scoo...
Patriks parallel pancakes
HungrySink
Pan 2
BatterSource
Pan 1
Balance Merge
nom nom nom
Patriks parallel pancakes
Flow<ScoopOfBatter, Pancake, NotUsed> fryingPan =

Flow.of(ScoopOfBatter.class).map(batter -> ne...
Patriks parallel pancakes
val pancakeChef: Flow[ScoopOfBatter, Pancake, NotUsed] =

Flow.fromGraph(GraphDSL.create() { imp...
Making pancakes together
HungrySink
Pan 3
BatterSource
Pan 1
Balance Merge
Pan 2
Pan 4
nom nom nom
Built in stages Flow stages
map/fromFunction, mapConcat,
statefulMapConcat, filter, filterNot,
collect, grouped, sliding, ...
But I want to
connect other
things!
@doggosdoingthings
A community for Akka Streams connectors
http://github.com/akka/alpakka
Alpakka
Alpakka – a community for Stream connectors
Existing Alpakka
MQTT
AMQP/
RabbitMQ
SSE
Cassandra
FTP/
SFTP
Kafka
AWS S3
File...
But my usecase is a
unique snowflake!
❄
❄
❄
GraphStage API
public class Map<A, B> extends GraphStage<FlowShape<A, B>> {

private final Function<A, B> f;

public final...
GraphStage API
class Map[A, B](f: A => B) extends GraphStage[FlowShape[A, B]] {



val in = Inlet[A]("Map.in")

val out = ...
What about distributed/reactive systems?
Kafka Stream Stream
Stream
Stream
cluster
The community
Mailing list:
https://groups.google.com/group/akka-user
Public chat rooms:
http://gitter.im/akka/dev develop...
Thanks for listening!
@apnylle
johan.andren@lightbend.com
Runnable sample sources (Java & Scala)
https://github.com/johana...
Prochain SlideShare
Chargement dans…5
×

Streaming all the things with akka streams

3 058 vues

Publié le

Slides from my Akka streams talk at Voxxeddays Zürich 2017-02-23

Publié dans : Ingénierie
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Awesome lonely girl looking for fun on webcam with the you now - www.xslideshare.usa.cc
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Presentation is good in general terms, but there is an inconsistency in page 43. It is stated that there is an existing Kafka connector in Alpakka community, but as per following references, there is no such a connector yet: https://github.com/akka/alpakka and http://developer.lightbend.com/docs/alpakka/current/index.html
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Streaming all the things with akka streams

  1. 1. Akka streams Streaming Johan Andrén Voxxeddays, Zürich, 2017-02-23 with All the things!
  2. 2. Johan Andrén Akka Team Stockholm Scala User Group @apnylle johan.andren@lightbend.com
  3. 3. Make building powerful concurrent & distributed applications simple. Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM Akka
  4. 4. Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community Complete Java & Scala APIs for all features What’s in the toolkit?
  5. 5. Reactive Streams
  6. 6. Reactive Streams timeline Oct 2013 RxJava, Akka and Twitter- people meeting “Soon thereafter” 2013 Reactive Streams Expert group formed Apr 2015 Reactive Streams Spec 1.0 TCK 5+ impls ??? 2015 JEP-266 inclusion in JDK9 Akka Streams, RxJava Vert.x, MongoDB, …
  7. 7. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols http://www.reactive-streams.org “
  8. 8. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols http://www.reactive-streams.org “
  9. 9. Stream processing Source Sink Flow
  10. 10. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols http://www.reactive-streams.org “
  11. 11. Asynchronous stream processing Source Sink (possible) asynchronous boundaries Flow
  12. 12. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments (JVM and JavaScript) as well as network protocols http://www.reactive-streams.org “
  13. 13. No back pressure Source Sink 10 msg/s 1 msg/s Flow asynchronous boundary
  14. 14. No back pressure Source Sink 10 msg/s 1 msg/s Flow asynchronous boundary OutOfMemoryError!!
  15. 15. No back pressure - bounded buffer Source Sink 10 msg/s 1 msg/s Flow buffer size 6 🗑 asynchronous boundary
  16. 16. Async non blocking back pressure Source Sink 1 msg/s 1 msg/s Flow buffer size 6 🗑 asynchronous boundary Hey! give me 2 more
  17. 17. Reactive Streams RS Library A RS library B async boundary
  18. 18. Reactive Streams “Make building powerful concurrent & distributed applications simple.”
  19. 19. Complete and awesome Java and Scala APIs (Just like everything in Akka) Akka Streams
  20. 20. Akka Streams in ~20 seconds: final ActorSystem system = ActorSystem.create();
 final Materializer materializer = ActorMaterializer.create(system);
 
 final Source<Integer, NotUsed> source =
 Source.range(0, 20000000);
 
 final Flow<Integer, String, NotUsed> flow =
 Flow.fromFunction((Integer n) -> n.toString());
 
 final Sink<String, CompletionStage<Done>> sink =
 Sink.foreach(str -> System.out.println(str));
 
 final RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);
 
 runnable.run(materializer); complete sources on github
  21. 21. Akka Streams in ~20 seconds: final ActorSystem system = ActorSystem.create();
 final Materializer materializer = ActorMaterializer.create(system);
 
 final Source<Integer, NotUsed> source =
 Source.range(0, 20000000);
 
 final Flow<Integer, String, NotUsed> flow =
 Flow.fromFunction((Integer n) -> n.toString());
 
 final Sink<String, CompletionStage<Done>> sink =
 Sink.foreach(str -> System.out.println(str));
 
 final RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);
 
 runnable.run(materializer); complete sources on github
  22. 22. Akka Streams in ~20 seconds: implicit val system = ActorSystem()
 implicit val mat = ActorMaterializer()
 
 val source = Source(0 to 20000000)
 
 val flow = Flow[Int].map(_.toString())
 
 val sink = Sink.foreach[String](println(_))
 
 val runnable = source.via(flow).to(sink)
 
 runnable.run() complete sources on github
  23. 23. Akka Streams in ~20 seconds: Source.range(0, 20000000)
 .map(Object::toString)
 .runForeach(str -> System.out.println(str), materializer); complete sources on github
  24. 24. Akka Streams in ~20 seconds: Source(0 to 20000000)
 .map(_.toString)
 .runForeach(println) complete sources on github
  25. 25. Numbers as a service final Source<ByteString, NotUsed> numbers = Source.unfold(0L, n -> {
 long next = n + 1;
 return Optional.of(Pair.create(next, next));
 }).map(n -> ByteString.fromString(n.toString() + "n"));
 
 
 final Route route =
 path("numbers", () ->
 get(() ->
 complete(HttpResponse.create()
 .withStatus(StatusCodes.OK)
 .withEntity(HttpEntities.create(
 ContentTypes.TEXT_PLAIN_UTF8,
 numbers
 )))
 )
 );
 
 final CompletionStage<ServerBinding> bindingCompletionStage =
 http.bindAndHandle(route.flow(system, materializer), host, materializer); complete sources on github
  26. 26. Numbers as a service val numbers =
 Source.unfold(0L) { (n) =>
 val next = n + 1
 Some((next, next))
 }.map(n => ByteString(n + "n"))
 
 val route =
 path("numbers") {
 get {
 complete( HttpResponse(entity = HttpEntity(`text/plain(UTF-8)`, numbers)) )
 }
 }
 val futureBinding = Http().bindAndHandle(route, "127.0.0.1", 8080) complete sources on github
  27. 27. recv buffer send buffer 🚚 🚚 🚚 🚚 🚚 🚚 🚚 Back pressure over TCP numbers TCP HTTP Server Client
  28. 28. recv buffer send buffer 🚚 🚚 🚚 🚚 🚚 🚚 🚑 Back pressure over TCP numbers TCP HTTP Backpressure Server Client
  29. 29. recv buffer send buffer 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚚 🚑 Back pressure over TCP numbers TCP HTTP Backpressure Backpressure Server Client
  30. 30. A more useful example complete sources on github final Flow<Message, Message, NotUsed> measurementsFlow =
 Flow.of(Message.class)
 .flatMapConcat((Message message) ->
 message.asTextMessage()
 .getStreamedText()
 .fold("", (acc, elem) -> acc + elem)
 )
 .groupedWithin(1000, FiniteDuration.create(1, SECONDS))
 .mapAsync(5, database::asyncBulkInsert)
 .map(written ->
 TextMessage.create("wrote up to: " + written.get(written.size() - 1))
 );
 
 final Route route = path("measurements", () ->
 get(() ->
 handleWebSocketMessages(measurementsFlow)
 )
 );
 
 final CompletionStage<ServerBinding> bindingCompletionStage =
 http.bindAndHandle(route.flow(system, materializer), host, materializer); Credit to: Colin Breck
  31. 31. A more useful example complete sources on github val measurementsFlow =
 Flow[Message].flatMapConcat(message =>
 message.asTextMessage.getStreamedText.fold("")(_ + _)
 )
 .groupedWithin(1000, 1.second)
 .mapAsync(5)(Database.asyncBulkInsert)
 .map(written => TextMessage("wrote up to: " + written.last))
 
 val route =
 path("measurements") {
 get {
 handleWebSocketMessages(measurementsFlow)
 }
 }
 
 val futureBinding = Http().bindAndHandle(route, "127.0.0.1", 8080)
  32. 32. The tale of the two pancake chefs HungrySink Frying Pan BatterSource Scoops of batter Pancakes nom nom nom asynchronous boundaries Roland Patrik
  33. 33. Rolands pipelined pancakes HungrySinkPan 2BatterSource Pan 1 nom nom nom
  34. 34. Rolands pipelined pancakes Flow<ScoopOfBatter, HalfCookedPancake, NotUsed> fryingPan1 =
 Flow.of(ScoopOfBatter.class).map(batter -> new HalfCookedPancake());
 
 Flow<HalfCookedPancake, Pancake, NotUsed> fryingPan2 =
 Flow.of(HalfCookedPancake.class).map(halfCooked -> new Pancake()); Flow<ScoopOfBatter, Pancake, NotUsed> pancakeChef =
 fryingPan1.async().via(fryingPan2.async()); section in docs
  35. 35. Rolands pipelined pancakes // Takes a scoop of batter and creates a pancake with one side cooked val fryingPan1: Flow[ScoopOfBatter, HalfCookedPancake, NotUsed] =
 Flow[ScoopOfBatter].map { batter => HalfCookedPancake() }
 
 // Finishes a half-cooked pancake
 val fryingPan2: Flow[HalfCookedPancake, Pancake, NotUsed] =
 Flow[HalfCookedPancake].map { halfCooked => Pancake() } 
 // With the two frying pans we can fully cook pancakes val pancakeChef: Flow[ScoopOfBatter, Pancake, NotUsed] =
 Flow[ScoopOfBatter].via(fryingPan1.async).via(fryingPan2.async) section in docs
  36. 36. Patriks parallel pancakes HungrySink Pan 2 BatterSource Pan 1 Balance Merge nom nom nom
  37. 37. Patriks parallel pancakes Flow<ScoopOfBatter, Pancake, NotUsed> fryingPan =
 Flow.of(ScoopOfBatter.class).map(batter -> new Pancake());
 
 Flow<ScoopOfBatter, Pancake, NotUsed> pancakeChef =
 Flow.fromGraph(GraphDSL.create(builder -> {
 final UniformFanInShape<Pancake, Pancake> mergePancakes =
 builder.add(Merge.create(2));
 final UniformFanOutShape<ScoopOfBatter, ScoopOfBatter> dispatchBatter =
 builder.add(Balance.create(2));
 
 builder.from(dispatchBatter.out(0)) .via(builder.add(fryingPan.async())) .toInlet(mergePancakes.in(0));
 builder.from(dispatchBatter.out(1)) .via(builder.add(fryingPan.async())) .toInlet(mergePancakes.in(1));
 
 return FlowShape.of(dispatchBatter.in(), mergePancakes.out());
 })); section in docs
  38. 38. Patriks parallel pancakes val pancakeChef: Flow[ScoopOfBatter, Pancake, NotUsed] =
 Flow.fromGraph(GraphDSL.create() { implicit builder =>
 import GraphDSL.Implicits._ 
 val dispatchBatter = builder.add(Balance[ScoopOfBatter](2))
 val mergePancakes = builder.add(Merge[Pancake](2))
 
 // Using two pipelines, having two frying pans each, in total using
 // four frying pans
 dispatchBatter.out(0) ~> fryingPan1.async ~> fryingPan2.async ~> mergePancakes.in(0)
 dispatchBatter.out(1) ~> fryingPan1.async ~> fryingPan2.async ~> mergePancakes.in(1)
 
 FlowShape(dispatchBatter.in, mergePancakes.out)
 }) section in docs
  39. 39. Making pancakes together HungrySink Pan 3 BatterSource Pan 1 Balance Merge Pan 2 Pan 4 nom nom nom
  40. 40. Built in stages Flow stages map/fromFunction, mapConcat, statefulMapConcat, filter, filterNot, collect, grouped, sliding, scan, scanAsync, fold, foldAsync, reduce, drop, take, takeWhile, dropWhile, recover, recoverWith, recoverWithRetries, mapError, detach, throttle, intersperse, limit, limitWeighted, log, recoverWithRetries, mapAsync, mapAsyncUnordered, takeWithin, dropWithin, groupedWithin, initialDelay, delay, conflate, conflateWithSeed, batch, batchWeighted, expand, buffer, prefixAndTail, groupBy, splitWhen, splitAfter, flatMapConcat, flatMapMerge, initialTimeout, completionTimeout, idleTimeout, backpressureTimeout, keepAlive, initialDelay, merge, mergeSorted, Source stages fromIterator, apply, single, repeat, cycle, tick, fromFuture, fromCompletionStage, unfold, unfoldAsync, empty, maybe, failed, lazily, actorPublisher, actorRef, combine, unfoldResource, unfoldResourceAsync, queue, asSubscriber, fromPublisher, zipN, zipWithN Sink stages head, headOption, last, lastOption, ignore, cancelled, seq, foreach, foreachParallel, onComplete, lazyInit, queue, fold, reduce, combine, actorRef, actorRefWithAck, actorSubscriber, asPublisher, fromSubscriber Additional Sink and Source converters {from,as}OutputStream, {from,as}InputStream, {as,from} javaCollector, javaCollectorParallelUnordered File IO Sinks and Sources fromPath, toPath mergePreferred, zip, zipWith, zipWithIndex, concat, prepend, orElse, interleave, unzip, unzipWith, broadcast, balance, partition, watchTermination, monitor Even more Framing, JSON framing, killswitch, BroadcastHub, MergeHub
  41. 41. But I want to connect other things! @doggosdoingthings
  42. 42. A community for Akka Streams connectors http://github.com/akka/alpakka Alpakka
  43. 43. Alpakka – a community for Stream connectors Existing Alpakka MQTT AMQP/ RabbitMQ SSE Cassandra FTP/ SFTP Kafka AWS S3 Files AWS DynamoDB AWS SQS JMS Azure IoT Hub TCP In Akka Actors Reactive Streams Java Streams Basic File IO Alpakka PRs AWS Lambda MongoDB* druid.io Caffeine IronMQ HBase
  44. 44. But my usecase is a unique snowflake! ❄ ❄ ❄
  45. 45. GraphStage API public class Map<A, B> extends GraphStage<FlowShape<A, B>> {
 private final Function<A, B> f;
 public final Inlet<A> in = Inlet.create("Map.in");
 public final Outlet<B> out = Outlet.create("Map.out"); private final FlowShape<A, B> shape = FlowShape.of(in, out); public Map(Function<A, B> f) {
 this.f = f;
 }
 public FlowShape<A,B> shape() {
 return shape;
 }
 public GraphStageLogic createLogic(Attributes inheritedAttributes) {
 return new GraphStageLogic(shape) {
 {
 setHandler(in, new AbstractInHandler() {
 @Override
 public void onPush() throws Exception {
 push(out, f.apply(grab(in)));
 }
 });
 setHandler(out, new AbstractOutHandler() {
 @Override
 public void onPull() throws Exception {
 pull(in);
 }
 });
 }
 };
 }
 } complete sources on github
  46. 46. GraphStage API class Map[A, B](f: A => B) extends GraphStage[FlowShape[A, B]] {
 
 val in = Inlet[A]("Map.in")
 val out = Outlet[B]("Map.out")
 override val shape = FlowShape.of(in, out)
 
 override def createLogic(attr: Attributes): GraphStageLogic =
 new GraphStageLogic(shape) {
 setHandler(in, new InHandler {
 override def onPush(): Unit = {
 push(out, f(grab(in)))
 }
 })
 setHandler(out, new OutHandler {
 override def onPull(): Unit = {
 pull(in)
 }
 })
 }
 } complete sources on github
  47. 47. What about distributed/reactive systems? Kafka Stream Stream Stream Stream cluster
  48. 48. The community Mailing list: https://groups.google.com/group/akka-user Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka Easy to contribute tickets: https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22 ~200 active contributors!
  49. 49. Thanks for listening! @apnylle johan.andren@lightbend.com Runnable sample sources (Java & Scala) https://github.com/johanandren/akka-stream-samples/tree/jfokus-2017 http://akka.io Akka

×