Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Alpakka - Connecting Kafka and ElasticSearch to Akka Streams

196 vues

Publié le

In order to work with Akka streams, we need a mechanism to connect Akka Streams to the existing system components. That is where Alpakka comes into the picture.
The Alpakka project is an open source initiative to implement stream-
aware, reactive, integration pipelines for Java and Scala.
It is built on top of Akka Streams and has been designed from ground
up to understand streaming natively and provide a DSL for reactive
and stream-oriented programming, with built-in support for
back pressure.

Publié dans : Logiciels
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Alpakka - Connecting Kafka and ElasticSearch to Akka Streams

  1. 1. Alpakka – Connecting Kafka and ElasticSearch to Akka streams By :- Anmol Sarna Software Consultant Knoldus Inc.
  2. 2. AGENDA ● Brief about Reactive Streams ● Introduction to Akka Streams ● What is Alpakka ● Types of Alpakka Connectors ● Alpakka ES and Kafka ● Demo
  3. 3. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. - Asynchrony is required to enable the parallel use of computing resources, and - Back pressure in needed to make sure the data producer doesn't overwhelm the data consumer and potentially bring down the entire system.
  4. 4. BackPressure
  5. 5. Solution (Pull Based backpressure)
  6. 6. Need for Reactive Streams? The goal of the Reactive Streams specification is to - create a standard for achieving statically typed, high-performance, low latency, and asynchronous streams of data with built-in non-blocking back pressure - Handling streams of data—especially “live” data whose volume is not predetermined—requires special care in an asynchronous system. - The most prominent issue is that resource consumption needs to be controlled such that a fast data source does not overwhelm the stream destination.
  7. 7. Akka Streams Definition : - Akka Streams, the Akka implementation of the Reactive Streams specification, is based on Actors as its mechanism for execution, distribution and resilience. - Akka Streams implement an asynchronous non-blocking back-pressure protocol.
  8. 8. Basic Concepts to Understand Akka Streams
  9. 9. Source : - Starting point of the stream - A processing stage with exactly one output, emitting data elements whenever downstream processing stages are ready to receive them. Flow : - The main processing building block - Every Flow instance has one input and one output value
  10. 10. Sink : - A Flow is not executed until we will register a Sink operation on it - It is a terminal operation that triggers all computations in the entire Flow - Exactly One input value Materializer : - A materializer makes actors execute a graph to produce those results. - At runtime the materializer walks the graph, looks at how it’s defined, and creates actors for all of the steps
  11. 11. Alpakka - End point for Akka Streams - The Alpakka project is an open source initiative to implement stream-aware, reactive, integration pipelines for Java and Scala. - It is built on top of Akka Streams and has been designed from ground up to understand streaming natively and provide a DSL for reactive and stream- oriented programming, with built-in support for backpressure.
  12. 12. About Alpakka - Open source initiative - Container project around multiple connectors. - Developer centric set of modules to connect systems using akka streams - Support for both java and scala
  13. 13. Alpakka Connectors Alpakka provides a large number of options to choose from when it comes to connectors. Some of them include : - Apache Cassandra - AWS S3 - Apache Kafka - HTTP - Elasticsearch - MongoDB and many more. (above 40)
  14. 14. - Alpakka Elasticsearch connector - Alpakka Kafka connector
  15. 15. Alpakka Elasticsearch connector The Alpakka Elasticsearch connector provides Akka Streams integration for Elasticsearch. Dependency : libraryDependencies += "com.lightbend.akka" %% "akka-stream-alpakka- elasticsearch" % "1.0-M2"
  16. 16. Alpakka ES connector can be used as : Source Flow Sink
  17. 17. Alpakka Kafka connector It was formerly known as Akka Streams Kafka and even Reactive Kafka.(by SoftwareMill) By combining Akka Streams with Kafka using Alpakka Kafka, we can build rich domain, low latency, and stateful streaming applications with very little infrastructure. Dependency : libraryDependencies += "com.typesafe.akka" %% "akka-stream-kafka" % "1.0-RC2"
  18. 18. Alpakka Kafka connector Example
  19. 19. References - https://doc.akka.io/docs/alpakka/current/ - https://www.slideshare.net/Lightbend/pakk-your-alpakka-reactive-streams- integrations-for-aws-azure-google-cloud - https://www.youtube.com/watch?v=L1FS2dyiwIg

×