In order to work with Akka streams, we need a mechanism to connect Akka Streams to the existing system components. That is where Alpakka comes into the picture. The Alpakka project is an open source initiative to implement stream- aware, reactive, integration pipelines for Java and Scala. It is built on top of Akka Streams and has been designed from ground up to understand streaming natively and provide a DSL for reactive and stream-oriented programming, with built-in support for back pressure.

1. Alpakka –
Connecting Kafka and ElasticSearch to Akka
streams
By :-
Anmol Sarna
Software Consultant
Knoldus Inc.

2. AGENDA
● Brief about Reactive Streams
● Introduction to Akka Streams
● What is Alpakka
● Types of Alpakka Connectors
● Alpakka ES and Kafka
● Demo

3. Reactive Streams
Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking
back pressure.
- Asynchrony is required to enable the parallel use of computing resources, and
- Back pressure in needed to make sure the data producer doesn't overwhelm the data consumer and
potentially bring down the entire system.

4. BackPressure

7. Solution
(Pull Based backpressure)

8. Need for Reactive Streams?
The goal of the Reactive Streams specification is to
- create a standard for achieving statically typed, high-performance, low latency, and asynchronous
streams of data with built-in non-blocking back pressure
- Handling streams of data—especially “live” data whose volume is not predetermined—requires special
care in an asynchronous system.
- The most prominent issue is that resource consumption needs to be controlled such that a fast data
source does not overwhelm the stream destination.

9. Akka Streams
Definition :
- Akka Streams, the Akka implementation of the Reactive Streams
specification, is based on Actors as its mechanism for execution, distribution
and resilience.
- Akka Streams implement an asynchronous non-blocking back-pressure
protocol.

10. Basic Concepts to Understand
Akka Streams

12. Source :
- Starting point of the stream
- A processing stage with exactly one output, emitting data elements whenever
downstream processing stages are ready to receive them.
Flow :
- The main processing building block
- Every Flow instance has one input and one output value

13. Sink :
- A Flow is not executed until we will register a Sink operation on it
- It is a terminal operation that triggers all computations in the entire Flow
- Exactly One input value
Materializer :
- A materializer makes actors execute a graph to produce those results.
- At runtime the materializer walks the graph, looks at how it’s defined, and creates
actors for all of the steps

14. Alpakka - End point for Akka Streams
- The Alpakka project is an open source initiative to implement stream-aware,
reactive, integration pipelines for Java and Scala.
- It is built on top of Akka Streams and has been designed from ground up to
understand streaming natively and provide a DSL for reactive and stream-
oriented programming, with built-in support for backpressure.

15. About Alpakka
- Open source initiative
- Container project around multiple connectors.
- Developer centric set of modules to connect systems using akka streams
- Support for both java and scala

16. Alpakka Connectors
Alpakka provides a large number of options to choose from when it comes to
connectors.
Some of them include :
- Apache Cassandra - AWS S3
- Apache Kafka - HTTP
- Elasticsearch - MongoDB
and many more. (above 40)

20. - Alpakka Elasticsearch
connector
- Alpakka Kafka
connector

21. Alpakka Elasticsearch connector
The Alpakka Elasticsearch connector provides Akka Streams integration
for Elasticsearch.
Dependency :
libraryDependencies += "com.lightbend.akka" %% "akka-stream-alpakka-
elasticsearch" % "1.0-M2"

22. Alpakka ES connector can be used as :
Source
Flow
Sink

23. Alpakka Kafka connector
It was formerly known as Akka Streams Kafka and even Reactive Kafka.(by
SoftwareMill)
By combining Akka Streams with Kafka using Alpakka Kafka,
we can build rich domain, low latency, and stateful streaming applications with
very little infrastructure.
Dependency :
libraryDependencies += "com.typesafe.akka" %% "akka-stream-kafka" % "1.0-RC2"

24. Alpakka Kafka connector Example

25. References
- https://doc.akka.io/docs/alpakka/current/
- https://www.slideshare.net/Lightbend/pakk-your-alpakka-reactive-streams-
integrations-for-aws-azure-google-cloud
- https://www.youtube.com/watch?v=L1FS2dyiwIg